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Overview 


Community colleges across the United States face a difficult challenge. On the one hand, they are 
“open access” institutions, with a mission to serve students from all backgrounds and at varying 
levels of college readiness. On the other hand, they must uphold high academic standards in order to 
maintain accreditation and prepare students for employment or transfer to four-year schools. How, 
then, can community colleges best serve students who want to learn but do not meet minimum 
academic standards? 

Chaffey College, a large community college located about 40 miles east of Los Angeles, began to 
wrestle with this question early in the twenty-first century. Under the auspices of a national demon- 
stration project called Opening Doors, Chaffey developed a program designed to increase probation- 
ary students’ chances of succeeding in college. Chaffey’ s program included a “College Success” 
course, taught by a counselor, which provided basic information on study skills and the requirements 
of college. As part of the course, students were expected to complete five visits to “Success Cen- 
ters,” where their assignments, linked to the College Success course, covered skills assessment, 
learning styles, time management, use of resources, and test preparation. 

In 2005, MDRC collaborated with Chaffey College to evaluate the one-semester, voluntary Opening 
Doors program. In 2006, the program was improved to form the two-semester Enhanced Opening 
Doors program, in which probationary students were told that they were required to take the College 
Success course. In MDRC’s evaluation of each program, students were randomly assigned either to 
a program group that had the opportunity to participate in the program or to a control group that 
received the college’s standard courses and services. This report presents the outcomes for both 
groups of students in the Enhanced Opening Doors evaluation for four years after they entered the 
study. The findings include: 

• The message matters — optional program activities had lower participation rates com- 
pared with required program activities. 

• Chaffey’s Enhanced Opening Doors program had positive short-term effects. When the 
two program semesters were complete, students in the program group had earned more credits 
than students in the control group and were nearly twice as likely as control group students to be 
in good academic standing. 

• Despite the program’s encouraging short-term effects, it did not meaningfully improve 
students’ long-term academic outcomes. Four years after the study began, program and con- 
trol group students had made similar' academic progress. Strikingly, during that time, only 7 per- 
cent of all students in the study had earned a degree or certificate. 

This report presents detailed findings from Chaffey’s Enhanced Opening Doors initiative, including 
the cost and cost-effectiveness of the program, and considers the implications of this research for 
designing services for probationary students in community college. 
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Preface 


Like many community colleges, Chaffey College enrolls large numbers of students who 
struggle academically and are eventually placed on probation for their poor academic perfor- 
mance. Many of these students ultimately leave college without earning a credential. “College 
Success” courses, which teach students the skills they need to navigate their way successfully 
through school, are one popular strategy used on numerous college campuses to improve 
students’ chances of succeeding after being placed on academic probation. 

Through a national demonstration project called Opening Doors, Chaffey’ s leaders de- 
veloped a program designed to increase probationary students’ chances of succeeding. The key 
feature of the program is a three-credit College Success course, which is taught by a college 
counselor. As part of the course, students are expected to visit the college’ s “Success Centers,” 
where individualized and group instruction in math, reading, and writing is available. 

This report describes Chaffey’ s program and its effects on students’ academic outcomes 
four years after students entered the program. This random assignment experimental study 
found that during the two semesters when students were eligible to participate in the program, 
they earned more credits, had a higher grade point average, and were more likely to be in good 
academic standing than students in a control group who did not participate in the program. Most 
of these findings, though, were a result of the credits earned in the College Success course, 
credits that are not applicable toward a degree and cannot be transferred to another institution. 
Nonetheless, the findings are notable because, in spite of adding the College Success class to 
their regular course load, students in the program kept pace with the control group (most of 
whom did not take the course) on academic outcomes. This finding suggested that when the 
program ended, the skills and study habits gained from the course might transfer to other 
classes. However, despite its early promise, the program’s effects were not sustained and did 
not translate into meaningful impacts on students’ academic success over the long term. 

These sobering findings are a reminder of the challenges faced by community colleges 
in general and students on probation in particular. While the Enhanced Opening Doors program 
may be a good start, a more intensive or longer-lasting initiative may be necessary to help 
probationary students beyond the short term. With the long-term results of the Opening Doors 
study at Chaffey now known, administrators, counselors, faculty, and researchers should begin 
to consider what can be done for probationary students in the future. 


Gordon L. Berlin 
President 
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Executive Summary 


Community colleges across the United States face a difficult challenge. On the one hand, they 
are “open access” institutions, with a clear mission to serve students from all backgrounds and 
at varying levels of college readiness. On the other hand, they must uphold high academic 
standards in order to maintain accreditation and adequately prepare students for employment or 
transfer to four-year colleges and universities. How, then, can community colleges best serve 
students who want to learn but do not meet minimum academic standards? 

Chaffey College, a large community college located about 40 miles east of Los Ange- 
les, began to wrestle with this question early in the twenty-first century. At the time, roughly 
one out of every five students enrolled at Chaffey was on probation for poor academic perfor- 
mance (meaning they had a cumulative grade point average, or GPA, below 2.0) or were 
making insufficient progress toward a degree (meaning that they failed to complete half or 
more of the credits they attempted). Under the auspices of a national demonstration project 
called Opening Doors, Chaffey developed a program that was designed to increase probation- 
ary students’ chances of succeeding in college. The program went through two iterations: 
Opening Doors, which began in fall 2005, and Enhanced Opening Doors, 1 which began the 
following year. The Enhanced Opening Doors program, which is the subject of this report, 
comprised three core components: 

• College Success course. Taught by a college counselor, this “guidance” 
course, which students in the program were told they had to take, was de- 
signed to help probationary students clarify their personal goals, understand 
college rules and regulations, and develop better study skills. A two-credit 
lecture course was linked to a one-credit workshop in which students would 
apply the principles covered in the lecture. The course’s credits counted to- 
ward full-time enrollment in the college and were included in students’ GPA, 
but did not count toward a degree or transfer to a four-year college or univer- 
sity. In addition, a voucher was provided to students in the program to cover 
the cost of College Success course books. A second-semester College Suc- 
cess course, which was not mandatory, was also offered to students in the 
program; however, only a minority of students enrolled in the second- 
semester College Success course. 


'At Chaffey, the second iteration of the program, called “Enhanced Opening Doors” in this report, was 
called “Opening Doors to Excellence.” 
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• Visits to the Success Centers. As part of the College Success course, stu- 
dents were expected to complete five visits to the college’s “Success Cen- 
ters,” where all Chaffey students could get extra help in reading, writing, and 
math. These visits also included assignments for students in the Enhanced 
Opening Doors program, linked to the College Success course, that covered 
the following topics: skills assessment, learning styles, time management, 
use of resources, and test preparation. 

• Improved counseling. The instructors of the College Success courses were 
expected to work with students during class time and to meet with them out- 
side of class to provide extra advising and counseling as needed. 

In order to determine the effectiveness of Chaffey’ s Enhanced Opening Doors program, 
MDRC — the nonprofit, nonpartisan organization responsible for launching and managing the 
Opening Doors demonstration — conducted a random assignment evaluation. MDRC randomly 
assigned 444 students either to a program group that was eligible to participate in the Enhanced 
Opening Doors program as described above or to a control group whose members could take a 
College Success course (though very few did), could visit the Success Centers on their own, and 
could access the college’ s standard counseling services. Random assignment ensures that both 
observed characteristics (for example, race and gender) and unobserved characteristics (for 
example, tenacity and motivation) are distributed similarly between the two research groups. 
Consequently, subsequent differences in academic outcomes — known as impacts — can be 
confidently attributed to the program, rather than to preexisting differences between the two 
research groups. 

Early results from the evaluation found that the program led to positive results during 
the two semesters that students were enrolled in the program (the “program semesters”). 2 This 
report extends follow-up on the Chaffey sample to four years after students entered the study to 
determine whether the Enhanced Opening Doors program continued to help students perform 
academically, persist in college, and earn college degrees at Chaffey or at other institutions to 
which they transferred during the study period. 

The key findings from this report are: 

• The message matters — optional program activities had lower partici- 
pation rates compared with required program activities. 


“Susan Scrivener, Colleen Sommo, and Herbert Collado, Getting Back on Track: Effects of a Community 
College Program for Probationary Students (New York: MDRC, 2009). 
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The original program, which is not the main focus of this report, offered students the 
chance to participate in the program’s College Success course, but did not require that they 
participate. In this first iteration of the program, only about half the students who were assigned 
to the program group enrolled in the class, believing that it was optional. 

In the Enhanced Opening Doors program, which began in the 2006-2007 academic 
year, students who were randomly assigned to the program group were informed that participa- 
tion in the first semester of Enhanced Opening Doors was required, and that they would not be 
able to register for other classes unless they signed up for the College Success course. Participa- 
tion in the first-semester College Success course jumped to 72 percent (from 52 percent in the 
original Opening Doors program). Although college administrators ultimately decided not to 
enforce the “requirement” for students who did not comply, evidence suggests that the change 
in message led to a higher rate of compliance, an important lesson from this project that may be 
applicable more broadly. 

• The short-term effects of Chaffey’s Enhanced Opening Doors program 
were encouraging. 

The Enhanced Opening Doors program was a two-semester intervention, as noted earli- 
er. Students in the program group and control group had significantly different experiences 
during the first semester of the program. Most program group students took the College Success 
course, whereas control group students did not, and program group students visited the Success 
Centers more often than did their control group counterparts. In theory, completing the College 
Success course and visiting the Success Centers will enable students to perform better academi- 
cally during the program period and to achieve longer-term success as a result of the skills and 
study habits they gain through these experiences. During the second semester of the program, in 
spring 2007, the experiences of program and control group students once again differed, 
although not nearly as dramatically as they had during the first program semester. 

When the two program semesters were complete, students in the program group had 
earned more credits than students in the control group and were more likely to have had a 
cumulative GPA of over 2.0. (See Table ES.l.) These positive program effects were partly 
driven by the College Success course, whose associated credits cannot be applied toward a 
degree and cannot be transferred to a four-year college or university. When credit accumulation 
is examined including only degree-applicable credits and credits earned through developmental 
education classes (which are not applicable toward a degree but count toward enrollment), and 
the College Success course is excluded, program and control group members were observed to 
have earned similar numbers of credits during the first two semesters of the study. However, 
most program group students took at least one semester of the College Success course during 
the first two program semesters in addition to their regular course load — so the fact that 
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The Opening Doors Demonstration 
Table ES.l 


Academic Outcomes at Chaffey College After Two Enhanced Opening Doors 

Program Semesters 

Chaffey College Four-Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Cumulative credits earned 

8.4 

5.7 

2 7 *** 

0.7 

Cumulative GPA higher than 2.0 a (%) 

35.7 

23.6 

Y2 \ *** 

4.3 

Cumulative degree-applicable GPA higher than 2.0 b (%) 

30.3 

23.2 

7.2 * 

4.2 

Ever in good academic standing 0 (%) 

30.4 

15.9 

14 5 *** 

4.0 

Sample size (n = 444) 

224 

220 




SOURCES: MDRC calculations from Chaffey College transcript and probation data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Cumulative measures are based on the two program semesters only. 

“Grades earned in all courses except for developmental courses are used in the calculation of grade point 
average (GPA). Students who do not have a GPA for this time period are included as zeros. 

b Grades earned only in degree-applicable courses, which exclude the College Success course, are used in the 
calculations of GPA. Students who do not have a GPA for this time period are included as zeros. 

“Sample members who were not enrolled were not considered to be in good academic standing. 


program group students did not fall behind control group members on degree-applicable and 
developmental credits may be viewed as a positive finding. In other words, despite having an 
additional three-credit class as part of their course load, program group students managed to 
keep pace with their control group counterparts on their other credits. 

In addition, the program had a positive impact on GPA for degree-applicable courses 
(that is, excluding the College Success course) during the two program semesters, suggesting 
that the positive program effects were not solely a result of the credits and grades in the College 
Success course. Finally, compared with their control group counterparts, students in the pro- 
gram group were nearly twice as likely to be in good academic standing at the end of the two- 
semester follow-up period. In sum, just after the program services were complete, program 
group students’ academic outcomes looked promising. 
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• Despite the program’s encouraging short-term effects, it did not mean- 
ingfully improve students’ long-term academic outcomes. 

This report presents results up to and including four years after students entered the 
study — an additional three years beyond the earlier short-term follow-up. At that point, it is 
expected that any positive short-term effect of the program would have translated into students 
making more progress toward a degree with respect to degree-applicable credits, which a 
student must earn in order to graduate, as well as developmental credits, which students are 
encouraged to complete prior to enrolling in certain degree-applicable courses. As shown in 
Table ES.2, the evidence from this study does not indicate that the program had significant, 
positive, long-term effects (compared with Chaffey College’s regular services) with respect to 
degree-applicable plus developmental credit accumulation, continued enrollment, or attainment 
of a degree or certificate. In general, students who were offered the opportunity to participate in 
the Enhanced Opening Doors program had academic outcomes that were similar to those of the 
students in the control group during the four-year follow-up period. After four years, program 
group members maintained their edge on total credits earned, but this impact mostly reflects the 
three-credit College Success course that students took during the first semester of the study, 
which does not help them move closer to earning a degree or certificate. Strikingly, four years 
after the study began, only 7 percent of all students in the study had earned a degree or certifi- 
cate, and 44 percent were still enrolled in school (with 21-22 percent still at Chaffey College). 

• The Chaffey College Enhanced Opening Doors program had a net cost 
of around $1,300 per program group member over the course of the two 
program semesters. 

The net cost of $1,300 per program group member represents the difference between 
the gross cost per program group member ($4,300) and the gross cost per control group member 
($3,000). One way to lower the cost of the program and potentially improve its overall cost- 
effectiveness would be to make the Enhanced Opening Doors program a one-semester, mandat- 
ed intervention. 

• When all credits are considered, including the College Success course, 
the cost per credit earned for the program group was slightly lower than 
for the control group. However, the cost-effectiveness of earning credits 
fades away when only degree-applicable and developmental credits are 
considered. 

Specifically, the cost per credit earned for program group members ($516 per credit) 
was 3 percent less expensive than the cost per credit earned for control group members ($530 
per credit). This finding is the result of program group members passing a higher percentage of 
their attempted credits compared with control members. However, when credits associated with 
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The Opening Doors Demonstration 
Table ES.2 


Primary Academic Outcomes, Enhanced Opening Doors, 
Four Years After Random Assignment 

Chaffey College Four-Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Number of semesters enrolled at any college 

4.18 

4.06 

0.12 

0.24 

Cumulative degree-applicable and developmental 

credits earned 2 ’ 

18.4 

17.1 

1.4 

1.9 

Enrolled at any college in year 4 or earned 

a degree or certificate (%) 

45.5 

45.5 

0.1 

4.7 

Enrolled at any college in year 4 

44.2 

44.1 

0.1 

4.7 

Earned a degree or certificate 

7.6 

6.4 

1.2 

2.4 

Sample size (n = 444) 

224 

220 




SOURCES: MDRC calculations from Chaffey College transcript, California Community College Chancellor's 
Office, and National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Enrollment and credits earned are based on data from fall 2006 through summer 2010. Degree and certificate 
data are from fall 2006 through spring 2010. 

“Includes credits earned in courses taken during the first four years of the study and excludes credits from the 
College Success course. If a student receives a letter grade in a course, this measure only includes credits in 
which the student earned a “C” or higher. If a student takes a class "pass/fail” or if the class is only offered 
“pass/fail," then this measure only includes a grade of “pass.” 
b At any public, two-year California college. 


the College Success course are excluded and only degree-applicable and developmental credits 
are considered, the relative cost-effectiveness of the program fades away. Specifically, the cost 
per degree-applicable and developmental credit earned for the program group ($797 per credit) 
was 26 percent more expensive than the cost per degree-applicable and developmental credit 
earned for the control group ($635 per credit). 


Conclusion 

Chaffey College is one of only a small percentage of community colleges in the United States 
that has been willing to subject one of its programs to a rigorous, random assignment evalua- 
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tion. Because of the college’s commitment to serving its students as well as possible, the staff 
members at Chaffey were eager to know how effective the Enhanced Opening Doors program 
was compared with the college’ s regular services for probationary students. Their willingness to 
participate in this study should be applauded, as it enables them to better serve their students 
and provides other college administrators, policymakers, and researchers with trustworthy 
information and evidence on which to base their decisions. Although there is not strong evi- 
dence that the Enhanced Opening Doors program has long-term outcomes that are significantly 
different from those produced by Chaffey’ s usual services, there are lessons that the college and 
policymakers can take away from this study. 

For example, the Opening Doors study at Chaffey College began as a study of one 
program (the Opening Doors program), and that program evolved into a stronger second 
program (the Enhanced Opening Doors program). A prior MDRC report describes the differ- 
ences between the two programs and their evolution in detail, but one of the main observations 
in the programs' development was a change from optional participation in certain program 
services to telling students that they were required to participate, which seemed to have effects 
on program participation and short-term program impacts . 3 Rates of participation in the 
College Success course were low in the original program, which led Chaffey administrators to 
require participation in the College Success course in the Enhanced Opening Doors program, 
and they told students that their registration would be blocked if they did not comply. This 
change in policy and messaging was associated with a large increase in program participation. 
It is sometimes the case that those individuals who are at the greatest risk of failure are also 
those who are the least likely to participate in programs that are designed to help them succeed. 
College administrators must weigh their desire to allow their students autonomy and decision- 
making power against the fact that doing so may reduce participation in the very programs that 
have been created for their benefit. Clearly, this is a difficult balance to achieve — requiring 
individuals to participate in a program without their buy-in can backfire, yet allowing complete 
flexibility, especially when dealing with students who have a low likelihood of success, may 
not be in the students’ best interest. 

A final lesson that has been emerging from a number of community college studies is 
that one- and two-semester interventions may not be sufficient to make a lasting difference . 4 
Students often do better while they receive interventions, but the impacts fade once the interven- 

3 Susan Scrivener, Colleen Sommo, and Herbert Collado, Getting Back on Track: Effects of a Community 
College Program for Probationary Students (New York: MDRC, 2009). 

4 See, for example, Susan Scrivener and Michael Weiss, More Guidance, Better Results? Three-Year Ef- 
fects of an Enhanced Student Sendees Program at Two Community Colleges (New York: MDRC, 2009); Evan 
Weissman, Kristin F. Butcher, Emily Schneider, Jedediah Teres, Herbert Collado, and David Greenberg, 
Learning Communities for Students in Developmental Math: Impact Studies at Queensborough and Houston 
Community Colleges (New York: MDRC, 201 1). 
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tions end. While it is not realistic — and probably not advisable — for interventions to last 
indefinitely, program operators might consider whether they can do more to help students make 
a smooth transition to regular college services once an intervention like Enhanced Opening 
Doors ends. For example, program operators might place greater emphasis on mapping out 
what courses students should take once the intervention ends and conducting periodic follow-up 
to make sure that students remain on track and have not encountered new barriers to success. 
Some experts have suggested that community colleges need to institute better policies and 
create clearer pathways that will help all students complete the requirements for earning a 
certificate or degree as quickly as possible . 5 Some promising new initiatives, such as the City 
University of New York’s Accelerated Study in Associate Programs (ASAP ) 6 and the Bill & 
Melinda Gates Foundation’s Completion by Design Initiative , 7 are aspiring to do just that, and 
may soon offer lessons on how it might be achieved. 


5 See, for example, Davis Jenkins, “Get with the Program: Accelerating Community College Students’ 
Entry into and Completion of Programs of Study,” CCRC Working Work Paper No. 32 (New York: Commu- 
nity College Research Center, Teacher’s College, Columbia University, 2011); Postsecondary Success Team, 
Completion by Design Concept Paper (Seattle: Bill & Melinda Gates Foundation, 2010). 

6 See www.cuny.edu/academics/programs/notable/asap/about.html. 

7 See www.completionbydesign.org. 
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Chapter 1 

Introduction 


“Today’s probationary student is tomorrow’s transfer student.” 

— Chaff ey College Administrator 

Community colleges across the United States face a difficult challenge. On the one hand, they 
are “open access” institutions, with a clear mission to serve students from all backgrounds and 
at varying levels of college readiness. On the other hand, they must uphold high academic 
standards in order to maintain accreditation and adequately prepare students for employment or 
transfer to four-year colleges and universities. How, then, can community colleges best serve 
students who want to learn but do not meet minimum academic standards? 

Chaffey College, a large community college located about 40 miles east of Los Ange- 
les, began to wrestle with this question early in the twenty-first century. At the time, roughly 
one out of every five students enrolled at Chaffey was on probation for poor academic perfor- 
mance (meaning that they had a cumulative grade point average, or GPA, below 2.0) or because 
they were making insufficient progress toward a degree (meaning that they had failed to 
complete 50 percent or more of the credits they had attempted). Under the auspices of a national 
demonstration project called Opening Doors, Chaffey developed a “College Success” course 
that taught probationary students how to set goals, manage time, and master other skills be- 
lieved to be important for academic success. The course also sent students to campus “Success 
Centers” where they received help in improving their reading, writing, or math skills. An 
evaluation by MDRC — the organization responsible for launching and managing the Opening 
Doors demonstration — found that the program yielded positive results. Specifically, after two 
semesters, students who had been assigned to participate in the program were nearly twice as 
likely to be off probation as students who had not been assigned to the program and did not take 
the College Success course. Students who were assigned to the program also earned more 
course credits and had a higher grade point average than students who were not in the program. 

MDRC’s first report on the Opening Doors program at Chaffey College, which ap- 
peared in 2009, followed up students in the study sample for two semesters after they entered 
the study, and found some promising early results. 1 This report extends follow-up on the 
Chaffey sample for four years after students entered the study to determine whether the program 
continued to help students perform satisfactorily, persist in college, and earn college degrees at 
Chaffey or other institutions. The results indicate that the early gains were generally short-lived. 


'Scrivener, Sommo, and Collado (2009). 
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After four years, students in the program were no more likely to be enrolled in college or to 
have earned a college degree than students who were not in the program. Students in the 
Opening Doors program maintained a small lead in the number of credits earned, but this was 
driven mostly by their completion of the College Success course, which did not count toward a 
degree and was not transferrable to four-year colleges and universities. 

The remainder of this chapter provides more detail on the Opening Doors demonstra- 
tion and the Chaffey program. This is followed by a brief discussion of the higher education 
system in California and what research has revealed about persistence and degree completion 
among community college students in the state and across the nation. It concludes with a 
summary of how the rest of the report is organized. 


The Opening Doors Demonstration 

As noted above, Chaffey College’s program for probationary students was part of a national 
demonstration project known as Opening Doors. MDRC launched the demonstration in re- 
sponse to two pressing problems: low rates of persistence among community college students 
and a dearth of reliable evidence about effective strategies to improve student retention and 
academic success. 2 After meeting with community college experts and students from around the 
country, MDRC identified three broad strategies expected to produce better student outcomes: 
(1) changes in curriculum or instruction that would help students master the basic skills needed 
to succeed in college; (2) increased financial aid that would give students an incentive to 
succeed and help them cover expenses; and (3) enhancements to academic counseling and other 
student services that would help students understand the “rules of the game” and remove 
barriers to success. MDRC then looked for states and community colleges that had promising 
program models or ideas that were consistent with one or more of these strategies, and that were 
interested in participating in a rigorous, random assignment evaluation. 

Four distinct programs were selected for the demonstration: 

• In a learning communities program at Kingsborough Community College 
in New York, incoming freshmen were placed into small groups and took 
all or most of their first-semester courses together. The learning communi- 
ties featured an English course (usually at a developmental, or remedial, 
level), a College Success course, and a standard college course like intro- 
ductory psychology that were connected by a common theme and integrat- 
ed course assignments. 


2 For an early history of the Opening Doors demonstration, see Brock and LeBlanc (2005). 
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• A perfomiance-based scholarship at Delgado Community College and the 
Louisiana Technical College-West Jefferson (both in the New Orleans area) 
provided a generous financial incentive to low-income parents if they stayed 
enrolled at least half time and maintained a “C” or better average. 

• In an enhanced student sendees program at Lorain County Community 
College and Owens Community College in Ohio, new and continuing stu- 
dents were assigned to academic counselors who had much lower-than- 
average caseloads and who worked as a team to deliver student services. 
Students received a modest stipend for completing at least two counselor 
visits per semester. 

• A College Success course that required students to get extra help in basic 
reading, writing, or math skills at Chaffey College in California targeted stu- 
dents who were on probation. 

Random assignment — long considered the “gold standard” for evaluations of welfare 
reform, health care, and other kinds of interventions — had rarely been done in community 
colleges before the Opening Doors demonstration. Students who met the colleges’ program 
eligibility criteria went through a process similar to a coin toss. At each college, about half the 
eligible students were assigned to a program group that received the intervention described for it 
above; the remaining students were assigned to a control group, which represented “business as 
usual” at the college — that is, what would be provided to students if the Opening Doors 
program did not exist. The strength of a such a design is that it ensures that the characteristics of 
students in the program and control groups are virtually identical at the beginning of the study, 
including characteristics that are easy to measure (for example, gender and prior educational 
history), as well as those that are more difficult to capture (such as motivation and tenacity). By 
tracking the program and control groups over time and comparing their outcomes, researchers 
can determine the impact of the program over and above regular college services — estimated 
as the difference between the outcomes in the program and control groups. 

MDRC released a series of reports on the effectiveness of the Opening Doors programs 
between 2006 and 2009. Most of these reports followed students for one or two years after 
random assignment, though the report on the enhanced student services program in Ohio 
tracked students for three years. 3 All the programs produced at least some positive impacts on 
credits earned or other measures. The largest effects were observed in the performance-based 
scholarship program in Louisiana, but the devastation caused by Hurricane Katrina in 2005 led 


3 For an overview of Opening Doors findings, see Scrivener and Coghlan (201 1). Full reports on individual 
Opening Doors programs are available at http://www.mdrc.org/project_31_2.html. 
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to temporary closure of the colleges in the study and complicated efforts to do longer-term 
follow-up on this sample. 4 

In 2010, MDRC received a grant from the U.S. Department of Education’s Institute of 
Education Sciences to conduct longer-term follow-up on students in the learning communities 
program at Kingsborough and the College Success course at Chaffey. These programs were 
singled out because of their initial positive results and because most of the students in the 
sample were relatively young (age 21 years or younger at the time of random assignment), 
which suggested that they were just beginning their postsecondary careers and would be good 
candidates for long-term follow-up. The primary research question was the same for both 
programs: would they help more students stay in school and earn degrees over the long run? As 
noted, this report tracks students in the Chaffey College sample for four years; MDRC will 
release a six-year follow-up study on the Kingsborough sample in 2012. 


Chaffey College and Opening Doors 

Chaffey College, one of the oldest community colleges in California, serves a region that has 
experienced enormous population growth and demographic change since the Second World 
War. In fall 2006, it enrolled nearly 18,000 students at its three campuses in Rancho Cucamon- 
ga, Chino, and Fontana 5 — all part of the vast “Inland Empire” that forms the nexus of Los 
Angeles, Riverside, and San Bernardino counties. The region is ethnically diverse, with a large 
and growing Hispanic population. 6 Over 40 percent of Chaffey’ s student population is Hispan- 
ic, 7 which designates it as an Hispanic Serving Institution eligible for Title V funding from the 
U.S. Department of Education. 

Chaffey College had a reputation for innovation long before Opening Doors. In 2000, 
the college launched the Basic Skills Transformation Project, which led to the creation of three 
“Success Centers” to help students improve their reading, writing, and math skills. The Success 
Centers are analogous to campus tutoring centers but offer much more than the average tutoring 
center. Led by full-time faculty, the Success Centers offer workshops and instruction to students 
on a variety of subjects, along with curriculum resources and instructional supports for faculty. 


4 See Richburg-Hayes et al. (2009). MDRC is coordinating with a team of researchers led by Christina 
Paxson at Princeton University on a long-term follow-up study of Opening Doors sample members in 
Louisiana who were affected by Hurricane Katrina. The study is focused on identifying how pre-hurricane 
resources and capacities — including mental and physical health, social networks, and economic resources — 
affect ability to adjust to a major life trauma. The study is funded by the National Institutes of Health (grant # 
R01HD046162). 

5 See http://nces.ed.gov/ipeds/datacenter/CDS.aspx. 

6 Johnson, Reed, and Hayes (2008). 

7 See http://nces.ed.gov/ipeds/datacenter/CDS.aspx. 
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Students can use the Success Centers on a drop-in basis or by appointment seven days a week, 
including early morning and evening hours on weekdays. The Centers are literally campus 
hubs, with hundreds of students coming through each day to study, use computers, attend 
workshops, and get help with homework. 

The Success Centers’ positive reputation led MDRC to approach Chaffey about the 
possibility of joining the Opening Doors demonstration. After some initial discussion, MDRC 
and Chaffey officials quickly determined that the Success Centers were not suited to a random 
assignment evaluation, since they were already embedded into campus life and were available 
to all students. Chaffey officials were intrigued, however, by the larger programmatic and 
research goals of the Opening Doors demonstration, and brought up the idea of developing an 
intervention that targeted students on probation. The college was alarmed by the sheer number 
of students in this category — approximately 3,500 when talks with MDRC began — and was 
committed to reducing this figure. The college also recognized that its current policy of sending 
students on probation a warning letter rarely changed students’ behavior or helped them 
improve. Rather than adopt a tougher stance, the college wanted to determine whether it could 
better utilize its Success Centers and counseling staff to help probationary students return to 
good standing and make progress toward a degree. 8 

The creation of Chaffey’ s Opening Doors program followed a series of meetings and 
retreats involving college administrators, institutional researchers, English and math faculty, 
Success Center coordinators, academic counselors, and financial aid staff at the college, along 
with representatives from MDRC. The earliest discussions centered on understanding the needs 
and characteristics of the probationary students. Relying heavily on data from the college’s 
Institutional Research office, what emerged was a picture of students who often had undefined 
goals, took too many classes or the wrong type of classes, did not understand college policies, 
and had poor study habits. The college’s Institutional Research office also reported data 
showing that probationary students were much less likely to frequent the Success Centers than 
higher-performing students. In response, the planning group devised an Opening Doors program 
that comprised three components: 

• College Success course. Taught by a college counselor, this one-semester 
“guidance” course was designed to help probationary students clarify their 
personal goals, understand college rules and regulations, and develop better 
study skills. A two-credit lecture course was linked to a one-credit workshop 
in which students would apply the principles covered in the lecture. The 
course’s credits counted toward full-time enrollment in the college and were 
included in students’ grade point averages, but did not count toward a degree 

8 For a complete discussion of the program history, see Scrivener, Sommo, and Collado (2009). 
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or transfer to a four-year institution. In addition, a voucher was provided to 
students in the program to cover the cost of College Success course books. 

• Visits to the Success Centers. As part of the College Success course, stu- 
dents were expected to complete nine visits to the college’ s Success Centers. 

These visits included assignments that were linked to the College Success 
course and covered skills assessment, learning styles, time management, use 
of resources, and test preparation. 

• Improved counseling. The instructor of the College Success course worked 
with students during class time and met with them outside of class as needed. 

Chaffey’s Opening Doors program went through two iterations. 9 The original program 
began with a small, 50-student pilot in spring 2005, and then proceeded to a full random 
assignment evaluation in fall 2005. Given the large number of students on probation, Chaffey 
had no difficulty recruiting nearly 900 students to participate. MDRC randomly assigned the 
students into a program group that was offered the services outlined above, or to a control group 
that was not allowed to participate in Opening Doors but could take advantage of other services 
on campus, including visiting the Success Centers and making appointments to see counselors. 10 
MDRC tracked both the program and control groups over four semesters and compared their 
enrollment status, grades, probationary status, and other outcomes. 

Soon after the fall 2005 semester ended — and before the impact results were available 
— Chaffey administrators, faculty, and staff got back together with MDRC to talk about how the 
program went and to consider possible improvements. At least three key critical implementation 
problems were identified. First and most important, roughly half the students who were assigned 
to the program group did not enroll in the College Success course, believing it to be optional. 
Second, many of the counselors who taught the College Success course were unsure about their 
roles, especially with regard to meeting with students outside of class. Third, the nine visits to the 
Success Centers were not well integrated into the College Success course, and struck some 
counselors and students as excessive. Perhaps owing to all these factors, MDRC’s impact 
evaluation ultimately found that the original Opening Doors program did not produce any added 
value, or impact, above and beyond the regular college services offered to the control group. 

The second iteration of the program, which became known as Enhanced Opening 
Doors — to convey a level of service that goes beyond the services offered through the original 


9 For a complete discussion of the two iterations of Chaffey’s Opening Doors program and early program 
impacts, see Scrivener, Sommo, and Collado (2009). 

l0 Some College Success courses were offered outside of the Opening Doors program in which control 
group students could enroll, but this happened only rarely. 
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Opening Doors program — began in the 2006-2007 academic year. 11 The most important 
change was that students who were randomly assigned to the program group were informed that 
participation was required, and that they would not be able to register for other classes unless 
they signed up for the College Success course. The content of the College Success course 
stayed about the same, but a second semester was added as an option for students who wanted 
or needed additional support. The counselors who taught the College Success course were more 
experienced than those in the original program and received clearer instructions about the 
importance of meeting with students outside of class. The counselors also worked with faculty 
in the Success Centers to develop more meaningful projects and assignments for students to 
improve their reading, writing, or math skills, and reduced the number of required visits to the 
Success Centers from nine to five. In the opinion of Chaffey College and MDRC staff, the 
Enhanced Opening Doors program operated much more smoothly than the original model and 
provided a sharp contrast to the college’s regular services and procedures for students on 
probation. (See Table 1.1.) 

Chaffey and MDRC staff felt that the revisions to the program were substantial enough 
to warrant another round of evaluation. Nearly 450 probationary students were recruited for and 
randomly assigned to Enhanced Opening Doors or the control group in fall 2006. The program 
group’s participation in the College Success course was vastly improved over the original 
model: nearly three-fourths (about 72 percent) of students participated in the first-semester 
College Success course, compared with 52 percent in the original program, and roughly one- 
third enrolled in the second-semester class. As shown in Table 1.2, the results were encourag- 
ing. At the end of the two semesters during which the program ran (the “program semesters”), 
students in the program group earned significantly more credits and were more likely to have a 
cumulative GPA higher than 2.0 compared with students in the control group. 12 These effects 
were largely driven by the College Success course, though the study found that the program 
also had a positive impact on GPA for degree-applicable courses. Finally, students in the 
program group were nearly twice as likely to be in good academic standing at the end of the 
two-semester follow-up period than students in the control group. 


Expectations for Long-Term Follow-Up 

As noted earlier, Chaffey College staff designed Opening Doors with the goal of reducing the 
number of students on probation and helping students get onto a path where they would eventu- 

11 At Chaffey, the second iteration of the program, called “Enhanced Opening Doors” in this report, was 
called “Opening Doors to Excellence.” 

I2 ln the original MDRC report. Getting Back on Track (Scrivener, Sommo, and Collado, 2009), GPA cat- 
egories were labeled as 0-1.9 and 2.0 or higher. The correct labels are 0-2.0 and greater than 2.0. Labels are 
corrected in this report. 
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The Opening Doors Demonstration 
Table 1.1 


Features of Enhanced Opening Doors and Regular College Services 
Chaffey College Four-Year Follow-Up Report 


Feature 

Enhanced Opening Doors Program 

Regular College Services 

Participation 

requirements 

Students were told that participation in the first- 
semester College Success course was required. 

No participation requirements. 

First-semester 
College Success 
course 

2-credit lecture linked with 1 -credit workshop 
designed to help probationary students 
develop skills needed for academic success; 
taught by experienced staff; almost three- 
fourths of program group took course; students 
expected to visit Success Centers. 

Students could take College 
Success course; very few 
control group members did so. 

Second-semester 
College Success 
course 

2-credit course designed to build upon first- 
semester course; all students who took first- 
semester course invited to participate; roughly 
one-third of program group took course; no 
expectation to visit Success Centers. 

Students could take College 
Success course; very few 
control group members did so. 

Success Centers 

Students in first-semester College Success 
course expected to visit centers 5 times; 
students chose which center(s) to visit; 
content of assignments integrated and timed 
with course material. 

Students could visit centers 
on their own; students in 
some developmental classes 
were required to do so; some 
control group members did so. 

Counseling 

Students in College Success courses received 
help from counselor in class, and many met 
with counselor outside of class; counselors 
generally worked proactively to identify and 
resolve issues. 

Students could access 
counseling on their own; 
caseload for counselors was 
roughly 1,500:1; counseling 
role was reactive. 


SOURCE: MDRC field research data. 


ally graduate from Chaffey or transfer to another college or university. The transfer goal is 
especially salient in California, where the Master Plan for Higher Education divides the state’s 
three public postsecondary education systems into three segments: the University of California 
(UC), which admits students who graduate in the top one-eighth of their high school graduating 
class; the California State Universities (CSUs), which admit students in the top one-third of 





The Opening Doors Demonstration 
Table 1.2 


Academic Outcomes at Chaff ey College After Two Enhanced Opening Doors 

Program Semesters 

Chaff ey College Four- Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Cumulative credits earned 

8.4 

5.7 

2 7 *** 

0.7 

Cumulative GPA higher than 2.0 a (%) 

35.7 

23.6 

12 i *** 

4.3 

Cumulative degree-applicable GPA higher than 2.0 b (%) 

30.3 

23.2 

7.2 * 

4.2 

Ever in good academic standing 0 (%) 

30.4 

15.9 

14 5 *** 

4.0 

Sample size (n = 444) 

224 

220 




SOURCES: MDRC calculations from Chaffey College transcript and probation data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Cumulative measures are based on the two program semesters only. 

a Grades earned in all courses except for developmental courses are used in the calculation of grade point 
average (GPA). Students who do not have a GPA for this time period are included as zeros. 

b Grades earned only in degree-applicable courses, which exclude the College Success course, are used in the 
calculations of GPA. Students who do not have a GPA for this time period are included as zeros. 
c Sample members who were not enrolled were not considered to be in good academic standing. 


their high school graduating class; and the community colleges, which admit all others. 13 
Because community colleges are less expensive and often closer to home than the UC schools 
or CSUs, many students who could qualify for the state’s four-year universities choose to attend 
community college first and later transfer. State budget cutbacks have also forced some univer- 
sity-eligible students to begin at community college because of space limitations. The UC 
schools and CSUs do not generally admit transfer students at the freshman or sophomore level, 
so community college students intending to transfer must complete a curriculum that includes 
60 transferable credits (about two years of full-time study) and at least one course in math and 
one in English. It is not necessary to earn an associate’s degree to transfer. 14 


13 Califomia State Department of Education (1960). 
14 Moore and Shulock (2010). 
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While the Master Plan suggests that community colleges offer a clear pathway to the 
state universities, recent research on community college completion and transfer rates indicate 
that relatively few students earn any type of credential or transfer to a four-year school. Six 
years after beginning at a community college in California, only 1 1 percent of degree-seeking 
students earn an associate’s degree, and another 5 percent earn an occupational certificate. 
Fifteen percent of degree-seeking students complete a transfer curriculum as defined by UC 
and CSU, and 23 percent transfer to either a state public university or a private institution. 15 
These figures are derived from state community college system, UC, and CSU databases and 
matched with data from the National Student Clearinghouse. 16 National surveys of entering 
community college students offer a picture that is only slightly better than the California data. 
Across the United States, 15 percent of degree-seeking community college students obtained 
an associate’s degree, and 6 percent earned a certificate from that institution within six years 
after beginning at a community college. An additional 32 percent had transferred to another 
public or private institution. 17 

These figures suggest that for students in the Chaffey study, academic success — as de- 
fined by degree completion and transfer — is more likely to be the exception than the rule. 
Unlike the state and national data cited above, the Chaffey sample is limited to students who 
were on probation at the time they entered the study, making them a particularly at-risk group. 
The follow-up period for this report is also four years rather than six, though all the students in 
the Chaffey sample had been enrolled in college for at least one year prior to random assign- 
ment. Chaffey and MDRC anticipated that Enhanced Opening Doors would equip at least some 
probationary students with the knowledge and skills they needed to help them recover from past 
challenges and go on to earn college degrees. As later chapters reveal, the analysis indicates that 
very few students attained this goal. 


Organization of This Report 

The subsequent chapters examine the educational trajectories of Chaffey students who were 
randomly assigned to the Enhanced Opening Doors program or a control group in fall 2006. 
Chapter 2 describes the study sample and data sources used to track students’ progress. Chapter 
3 reviews the participation data for the fall 2006 program and describes the program’ s impacts 
during the program year and the year subsequent to the program. Chapter 4 takes the long-term 
view, covering impacts on various educational measures over four years. Chapter 5 presents 
information on program costs, and Chapter 6 gives an overall summary and outlines implica- 
tions for policymakers and community college practitioners. 


15 Moore and Shulock (2010). 

16 See Chapter 2 for more information on these databases. 
17 Radford, Berkner, Wheeless, and Shepherd (2010). 
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Chapter 2 


Data Sources and Sample Description 

As discussed in Chapter 1, the evaluation at Chaffey College uses a random assignment design 
to estimate the effects of the Enhanced Opening Doors program compared with Chaffey’ s 
regular classes and services offered to probationary students. This chapter describes how 
students became part of the research sample and presents some characteristics of the sample 
members. It also includes key information about the data sources used in this report. 


Identifying, Recruiting, and Randomly Assigning Students 

Chaffey targeted the Enhanced Opening Doors program to students who met the following 
eligibility criteria: 

• Were on academic or progress probation (see below) 

• Had earned fewer than 35 credits toward a degree or credential 

• Did not have an associate’s degree (or higher) from an accredited college or 
university 

• Had a high school diploma or General Educational Development (GED) cer- 
tificate 

• Were between 18 and 34 years of age 


Students at Chaffey are placed on academic probation if they have attempted at least 12 
credits (since starting college) and have a cumulative grade point average (GPA) below 2.0 
(“C”). Students are placed on progress probation if they have attempted at least 12 credits and 
have not completed at least half of the credits they attempted. The Enhanced Opening Doors 
program targeted students on either type of probation. 

Chaffey identified individuals who were eligible for the study using the college’s stu- 
dent database. The college sent letters to eligible students notifying them of their probationary 
status and communicating the college’s desire to help them succeed academically. The letter 
stated, “Failure to improve your academic status may subject you to dismissal from Chaffey 
College.” The letter informed students that they were required to attend one of the probation 
orientation sessions that were offered and that if they did not attend a session before their 
registration date, they would not be able to register for classes. College staff made follow-up 
phone calls to students who did not show up for an orientation session, reiterating this message. 
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At the orientation sessions, Chaffey staff explained the purpose of the study and what 
students needed to do to improve their academic standing. If students agreed to participate in the 
study, staff obtained their written consent and collected baseline information (discussed below). 
Staff transmitted students’ information to MDRC over a secure Web site, received students’ 
research group designation from MDRC (that is, whether they were assigned to the program or 
control group), and informed the students of their research group. 

The students who were assigned to the program group were scheduled to meet with a 
program counselor. Staff told program group students that they were required to register for the 
College Success course or they would not be permitted to register for any other courses. After 
sample intake ended, Chaffey decided not to block students' registration if they did not register 
for the College Success course because of concern about enrollment rates at the college. 
However, program group students reported that they believed that they were required to take 
the course. 

After random assignment, the students in the control group attended a brief workshop, 
during which they were told in more detail what they needed to do to improve their academic 
standing (for example, work to get their GPA up to a 2.0 or better). There they were also 
encouraged to schedule an appointment with a college counselor. Control group members were 
then able to seek services on their own. 

Between March and August 2006, 444 students were randomly assigned, with 224 as- 
signed to the program group and 220 assigned to the control group. The college recruited the 
fall 2006 cohort of students in two different rounds. The first round targeted students who were 
on probation at the end of fall 2005, and the second round targeted students who were on 
probation at the end of spring 2006. The college identified students who met all the study 
eligibility criteria and seemed likely to still be on probation when the intervention began. 1 


Characteristics of the Sample 

Table 2.1 shows some characteristics of the Enhanced Opening Doors sample members at 
Chaffey based on the Baseline Information Form (BIF), a questionnaire they completed just 
before they were randomly assigned. In addition, the table includes some measures of prior 
academic progress at Chaffey, based on Chaffey transcript data. The table shows the character- 
istics for the full sample — that is, the program group and control group combined. 


Tor a more detailed description of the recruitment and intake process, see Scrivener, Sommo, and Collado 
( 2009 ). 
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The Opening Doors Demonstration 
Table 2.1 

Selected Characteristics of Sample Members at Baseline, 
Enhanced Opening Doors 


Chaff ey College Four- Year Follow-Up Report 


Characteristic 

Full 

Sample 

Gender (%) 

Male 

38.3 

Female 

61.7 

Age in years (%) 

18-20 

60.6 

21-25 

29.5 

26-30 

5.6 

31-34 

4.3 

Marital status (%) 

Married 

5.8 

Unmarried 

94.2 

Race/ethnicity 3 (%) 

Hispanic/Latino 

54.2 

Black, non-Hispanic 

12.2 

White, non-Hispanic 

21.8 

Asian or Pacific Islander 

6.0 

Other 

5.8 

Has one child or more (%) 

10.8 

Household receiving any government benefits' 3 (%) 

12.1 

Financially dependent on parents (%) 

51.3 

Ever employed (%) 

92.6 

Currently employed (%) 

74.6 

Diplomas/degrees earned 0 (%) 

High school diploma 

95.7 

General Educational Development (GED) certificate 

3.6 

Occupational/technical certificate 

6.3 

Date of high school graduation/GED certificate receipt (%) 

Dining the past year 

32.2 

Between 1 and 5 years ago 

50.9 

More than 5 years ago 

17.0 


(continued) 


13 





Table 2.1 (continued) 


Full 

Characteristic Sample 

Main reason for enrolling in college (%) 

To complete a certificate program 5.8 

To obtain an associate's degree 24.8 

To transfer to a 4-year college/university 62.9 

To obtain/update job skills 2.7 

Other 3.9 

First person in family to attend college (%) 30.8 

Working personal computer in home (%) 90.3 

Owns or has access to a working car (%) 88.9 

Language other than English spoken regularly in home (%) 35.6 

U.S. citizen (%) 93.0 

Respondent born outside U.S. d (%) 1 1.0 

Respondent or respondent’s parent(s) born outside U.S. d (%) 50.3 

Prior academic nroarcss at Chaffev College 0 

Number of semesters enrolled 3.56 

Number of credits earned 13.5 

Cumulative GP A 1 .4 

On probation (%) 98.9 

One semester on probation 48.4 

Two semesters or more on probation 50.5 


Sample size 444 


SOURCES: MDRC calculations using Baseline Information Form (BIF) and Chaffey College 
transcript and probation data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

a Respondents who said they are Hispanic/Latino and chose a race are included only in the 
Hispanic/Latino category. Respondents who are not Hispanic/Latino and chose more than one race are 
considered multiracial. “Other” includes American Indian/ Alaskan Native, multiracial, and other. 

b Benefits include Unemployment/Dislocated Worker benefits. Supplemental Security Income (SSI) 
or disability, cash assistance or welfare, food stamps, and Section 8 or public housing, 
distributions may not add to 100 percent because categories are not mutually exclusive. 
d “U.S.” includes Puerto Rico. 

e As of the start of the program. Includes information from Chaffey from fall 1999 through summer 
2006. 
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About 62 percent of the sample members are women. The large majority were 25 years 
of age or younger (90. 1 percent), and most were 20 years of age or younger (60.6 percent) at the 
time of random assignment. Just over half of the sample identified themselves as Hispan- 
ic/Latino. About one in five identified themselves as white (non-Hispanic), and one in eight 
reported their race as black (non-Hispanic). Nearly all were unmarried (94.2 percent) and very 
few had children (10.8 percent). Three-fourths of the sample were employed at the time of study 
intake and about half said they were financially dependent on their parents. 

Nearly all sample members had previously earned a high school diploma (95.7 percent). 
When asked what the main reason for enrolling in Chaffey College was, close to two-thirds said 
to transfer to a four-year college or university (62.9 percent). Less than a third reported being 
the first in their family to attend college. 

As mentioned earlier, this program was targeted to students who were on probation. 
That means that these students had previously been enrolled at Chaffey and had experi- 
enced substantial academic difficulties. Prior to the start of the program, sample members 
had been enrolled at Chaffey for an average of 3.56 semesters, earned an average of 13.5 
credits, and had an average cumulative GPA of 1.4. Virtually all sample members were still 
on probation as of the start of the program semester, and about half had been on probation 
for two or more semesters. 

Appendix Table A.l shows the same characteristics as those presented in Table 2.1 for 
the full Enhanced Opening Doors sample, the program group, and the control group. One or 
more asterisks in the rightmost column of the table indicates that the difference between the 
proportion of program and control group members with that characteristic is statistically 
significant — that is, it most likely did not arise by chance. There were very few statistically 
significant differences in baseline characteristics between the groups. 2 

Table 2.2 compares the research sample, nearly all of whose members were on proba- 
tion when they entered the study (that is, at the time of random assignment), with the degree- 
seeking student body at Chaffey on a few demographic characteristics. 3 The sample members 
reflect the gender make-up of the campus. The research participants are younger, however, than 
the overall student body. Close to half of the research sample was younger than 20 years of age 
compared with less than a third of the Chaffey student body. Both the student body and the 


2 In addition, an omnibus test was conducted to assess whether overall systematic differences in baseline 
characteristics were observed between the two research groups. The model's likelihood ratio yielded a p- value 
of 0.38. Convention suggests that this probability of differences occurring by chance is large enough that these 
differences can be ignored in the analysis. 

3 Due to data limitations, the statistics on age for the Chaffey student body are based on the entire fall 2006 
student body, not just degree- seeking students. 
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The Opening Doors Demonstration 
Table 2.2 

Selected Characteristics of Chaffey College Student Body and 
Enhanced Opening Doors Study Sample 


Chaffey College Four- Year Follow-Up Report 


Characteristic (%) 

Chaffey 

Student 

Body 

Study 

Sample 

Gender 

Male 

39.1 

38.3 

Female 

60.9 

61.7 

Age in years 3 

Less than 20 

30.3 

45.7 

20-24 

34.4 

41.7 

25 and older 

35.3 

12.6 

Race/ethnicity 

Hispanic 

42.6 

54.2 

Black, non-Hispanic 

12.9 

12.2 

White, non-Hispanic 

27.6 

21.8 

Asian or Pacific Islander 

8.1 

6.0 

Other b 

8.9 

5.8 

Sample size 

11,811 

444 


SOURCES: U.S. Department of Education, National Center for Education Statistics, Integrated 
Postsecondary Education Data System (IPEDS), and MDRC calculations using Baseline Information Form 
(BIF) data. 

NOTES: IPEDS data on undergraduate degree-seeking students from fall 2006. 

Rounding may cause slight discrepancies in sums and differences. 

“Data on age are based on the entire undergraduate student population. 

b Race/ethnicity categories available in the IPEDS data include Hispanic, non-Hispanic white, non- 
Hispanic black. Native American or Alaskan Native, Asian or Pacific Islander, and unknown race or 
nonresident alien. An “Other” category was created that combined the unkown race. Native American or 
Alaskan Native, and nonresident alien categories for IPEDS data. The “Other” category in the BIF data 
includes students who marked “Other race,” Native American or Alaskan Native, or more than one race on 
the BIF. 
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research sample were diverse, but the research sample included a higher proportion of Hispanic 
students. The research sample, which consisted of students on probation who met other program 
eligibility criteria, should not be considered representative of the broader Chaffey student body. 


Data Sources 

To study the Enhanced Opening Doors program, the analyses presented in this report rely on 
several data sources, described below. All data sources were provided for both the program and 
control groups. 4 

Baseline Data 

As mentioned above, just before students were randomly assigned to the study groups, 
they completed a questionnaire called the Baseline Information Form, which collected demo- 
graphic and other background information. Baseline data are used in this report to describe the 
sample. 


Chaffey College School Records Data 

Chaffey provided MDRC with transcript, probation, and Success Center participation 
data. These data are specific to Chaffey College and do not capture what sample members 
may have been doing at other colleges or universities. Each of these is discussed in more 
detail below. 

Transcript data include various academic outcomes for courses taken at Chaffey, such 
as courses for which students registered, number of credits earned, and course grades. They also 
include degrees and certificates earned at Chaffey. The analyses for this report include data 
through the spring 2010 semester, which represents four years of follow-up (the year in which 
the program ran, or the “program year,” plus three “postprogram years”). 

Probation data include the sample members’ probation status as of the end of a semes- 
ter, and are used to determine whether they are on probation in a given semester. This report 
presents probation data through the spring 2008 semester, which is two years of follow-up (the 
program year, plus one postprogram year). 5 In addition, these data are used to describe the 
sample members’ probation status at the start of the program. 


4 For a more detailed description of how these data sources were used to create measures, see Appendix B. 

5 In this report, probationary status is not tracked beyond two years for several reasons. First, only students 
who are currendy enrolled are considered eligible to be in good academic standing. Since only 37 percent of all 
students in the research sample enrolled at Chaffey during the fourth semester of the study, probationary status 

(continued) 
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Success Center participation data include the incidence of students’ visits to the centers. 
Students using these services sign in and out with a computerized swipe-card. The Success 
Center data are presented for two years following the start of the program (that is, the program 
year plus one postprogram year). 

Chaffey College Financial Data 

To determine the cost of operating the Enhanced Opening Doors program, Chaffey 
provided MDRC with budget data for the program. Chaffey worked with MDRC to clarify 
descriptions of spending to ensure that all financial data were presented accurately. To estimate 
the control group cost, the Chaffey operating budget for fiscal year 2006-2007 was pulled from 
the college’s Web site. The cost-effectiveness study is presented in Chapter 5. 

National Student Clearinghouse Data 

The National Student Clearinghouse, a nonprofit organization, collects and distributes 
enrollment, degree, and certificate data from more than 3,000 colleges that enroll more than 90 
percent of the nation’s college students. 6 The Clearinghouse data are used to provide infor- 
mation about students in the study who may have attended a postsecondary institution other 
than Chaffey. 

California Community College Chancellor’s Office Data 

The California Community College Chancellor’s Office (CCCCO) collects student- 
level data from the California Community College system, which comprises 72 districts and 
112 colleges, and enrolls more than 2.9 million students. CCCCO provided academic data to 
MDRC from all participating schools that a sample member attended. These data cast a wider 
net than the Chaffey transcript data, as they capture the academic progress of sample members 
in the California Community College system. However, they do not capture academic gains 
made outside of the system, such as at four-year public or private institutions, or schools 
outside of California. 


becomes highly correlated with enrollment status as time passes. Enrollment status is tracked for the entirety of 
the follow-up period. Second, probation data are only available at Chaffey, so as more students attend or 
transfer to other institutions, this measure becomes less meaningful. 

6 See www.studentcleaiinghouse.org. 
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In addition, the CCCCO’s Data Mart Tool (available on its Web site) was used to pull 
information on the number of full-time equivalent (FTE) students for the 2006-2007 academic 
year at Chaffey College for the cost-effectiveness study. 7 

Field Research 

During the operation of the Enhanced Opening Doors program, MDRC staff visited 
Chaffey to conduct field research and interviewed many college administrators and staff 
involved with the program. The interviews provided information about the operation of the 
program and about the key differences between the program and the standard college courses 
and services available to the members of the study’s control group. MDRC staff also conducted 
a small number of individual student interviews and group discussions. 


7 FTE is a measurement used to standardize student workload so that all students are comparable, even 
though they may study a different number of hours per semester (or in a given time period). One FTE, for 
example, is equal to one student enrolled full-time for one academic year. 
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Chapter 3 

Program Participation and Short-Term Impacts 


This chapter begins by describing the participation in and use of the Enhanced Opening Doors 
program services by the fall 2006 cohort of students at Chaffey College. Participation in the 
College Success course and visits to the Success Centers are described during the academic year 
in which students were eligible to participate in the program (fall 2006 and spring 2007) as well 
as during two postprogram semesters (fall 2007 and spring 2008). Program services usage 
focuses on enrollment in the College Success course and visits to the Success Centers — the 
two key features of Chaffey’ s Enhanced Opening Doors program. The program theory posits 
that higher rates of enrollment in the College Success course combined with increased Success 
Center usage will lead to increased academic success. In order to document whether the 
program induced increased use of Chaffey’ s services for students on probation, enrollment in 
the College Success course and visits to the Success Centers are reported here for both the 
program group and the control group. It is the differences in experiences between the two 
groups that are likely to have brought about any observed changes in — or impacts on — 
academic outcomes. 

Following a description of program participation, the chapter focuses on the impact of 
the sample members’ different experiences on various measures of academic progress. The 
chapter summarizes the highlights from an earlier MDRC report, 1 which focused on the 
program’s effectiveness during the two semesters in which the program operated. 2 In addition 
to presenting the effects during the two program semesters, the chapter includes new evidence 
of the program’s cumulative effects after two postprogram semesters. 3 Since so few students 
enrolled during the summer, the summer semesters are included in the spring semester results. 
Figure 3.1 maps the time period described in this chapter, with black arrows representing the 
program semesters. Following this chapter, Chapter 4 presents the long-term (four years after 
students entered the study) impact of the Enhanced Opening Doors program on a selected set 
of outcomes. 


'Scrivener, Sommo, and Collado (2009). 

2 Unlike in the original report and Chapter 1 of this report, these results now include the summer semester 
as well. 

3 The summer term is included in the cumulative results. 
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The Opening Doors Demonstration 


Figure 3.1 


Data Collection Timeline, First Four Semesters of Enhanced Opening Doors Study 


Chaffey College Four- Year Follow-Up Report 


Semester 1 data Semester 2 data Semester 3 data Semester 4 data 



Fall 

’07 



Spring 

’08 



Summer 

’08 



Program semester 


| — | Postprogram semester 


NOTE: Data for semester 1 are from fall 2006; data for semester 2 are from spring 2007 and summer 2007; 
data for semester 3 are from fall 2007; and data for semester 4 are from spring 2008 and summer 2008. 

Participation in Program Services 

The two-semester Enhanced Opening Doors program operated during the fall 2006 and spring 
2007 semesters. Prior to the start of the fall 2006 semester, 444 students signed up to participate 
in the MDRC study; 224 were randomly assigned to the program group and the remaining 220 
were randomly assigned to the control group. Students assigned to the program group were told 
that they were required to participate in the program. 

During the first program semester of the study (fall 2006), the Enhanced Opening 
Doors requirements included (1) completion of an education plan that identified goals and a 
timeline with steps to achieve those goals; (2) enrollment in a three-credit College Success 
course; 4 and (3) a minimum of five visits to the Success Centers, including completion of five 


4 Credits were not applicable toward a degree and were nontransferable. The three-credit course comprised 
a two-credit lecture course linked to a one-credit workshop in which students would apply the principles 
covered in the lecture. 


22 


assignments at the Success Centers that counted toward a student’s grade in the College 
Success course. 

During the second program semester of the study (spring 2007), a two-credit College 
Success course was offered to program group students on a voluntary basis. 5 However, visiting 
Success Centers was not an explicit expectation of the College Success course or the program 
more broadly, although Chaffey administrators hoped that habits formed during the first 
program semester course would continue during the second program semester and beyond. 
Summarized here are some key quantifiable measures of program participation. 6 

As shown in Table 3.1, approximately 72 percent of all program group students regis- 
tered for the College Success course during the first program semester, and around 29 percent of 
all program group students registered for the College Success course during the second program 
semester. Most students who enrolled in the second semester Success Course had also enrolled 
in the first semester Success Course. In the postprogram semesters (the third and fourth semes- 
ters), very few program group students enrolled in a College Success course, although some 
control group students (4.5 percent in the third semester) did enroll in this type of course. If the 
College Success course has a positive effect on students, the fact that a small percentage of 
control group students were able to take the course in the postprogram semesters may slightly 
dilute estimated program effects beyond the first year of the study. 

Program group students visited the Success Centers significantly more often than their 
control group counterparts, particularly during the first program semester, when visits were an 
integrated part of the required College Success course. During the first program semester, 
around 69 percent of program group members visited a Success Center at least once, with the 
average number of visits being 5.5 times. 7 In contrast, during the same semester, only 32 percent 
of control group members visited a Success Center at least once, with the average number of 
visits being 1.8 times. 

During the second program semester (spring 2007), visiting the Success Centers was 
not an explicit expectation of the College Success course. Nonetheless, college administrators 
were optimistic that the habits formed during the first semester would translate into behaviors in 
the second semester and beyond. Despite these hopes, Success Center usage dropped to only 29 
percent of program group students during the second program semester. While the decrease in 
Success Center usage was dramatic, usage by program group students still exceeded usage by 


5 Credits for this course, like those for the first semester course, were non-degree-applicable and non- 
transferable. 

6 A more detailed description of the program services appears in Scrivener, Sommo, and Collado (2009). 

7 This average includes the full sample; that is, it includes zeros for the 3 1 percent of students who never 
visited the Success Centers. 
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The Opening Doors Demonstration 
Table 3.1 

Enrollment, Registration in College Success Course, and Visits to Success Center, 
Enhanced Opening Doors Study, First Four Semesters 


Chaffey College Four-Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Enrolled 2 (%) 

First semester 

85.3 

80.9 

4.4 

3.6 

Second semester 

67.9 

61.8 

6.1 

4.5 

Third semester 

41.5 

47.3 

-5.8 

4.7 

Fourth semester 

36.6 

36.8 

- 0.2 

4.6 

Registered for a College Success course 15 (%) 

First semester 

72.3 

0.5 

71 9 *** 

3.1 

Second semester 

29.0 

0.0 

29 o *** 

3.1 

Third semester 

2.7 

4.5 

-1.9 

1.8 

Fourth semester 

0.4 

0.9 

-0.5 

0.8 

Ever visited a Success Center (%) 

First semester 

68.8 

31.8 

37 o *** 

4.4 

Second semester 

28.6 

20.5 

8.1 ** 

4.1 

Third semester 

20.5 

20.5 

0.1 

3.8 

Fourth semester 

16.5 

15.9 

0.6 

3.5 

Number of visits to Success Center 

First semester 

5.5 

1.8 

2 7 *** 

0.7 

Second semester 

2.5 

1.3 

l 2 *** 

0.5 

Third semester 

2.0 

1.5 

0.5 

0.5 

Fourth semester 

1.5 

1.2 

0.3 

0.4 

Sample size (n = 444) 

224 

220 




SOURCES: MDRC calculations using Chaffey College transcript and Success Center participation data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

a The summer semesters are included as part of the spring semester results. 

b Registered for any Opening Doors College Success course. Opening Doors College Success courses include 
Guidance 506 College Success. Guidance 511 College Success Seminar, and Guidance 592D Special Topics; 
Guidance. 
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control group students, of whom only 21 percent ever visited a Success Center during the 
second semester of the study. Beyond the second semester, once the program was complete, use 
of the Success Centers continued to drop and rates of usage were similar for program and 
control group students. 

Part of the decreased Success Center usage from semester to semester simply reflects a 
decrease in the percentage of students enrolling at Chaffey overall; however, the rates of usage 
among program group members who enrolled at Chaffey dropped steeply between the first and 
second program semesters, from 81 percent (68.8/85.3) to only 42 percent (28.6/67.9). Similarly 
large drop-offs were not observed for control group students, who started at a much lower rate 
of Success Center usage. This suggests that once students were no long expected to visit the 
Success Centers, usage largely returned to typical (lower) levels. 

The participation data show that, generally speaking, program and control group stu- 
dents had significantly different experiences during the first program semester of the Enhanced 
Opening Doors program. Most program group students took the College Success course, 
whereas control group students did not, and program group students visited the Success Centers 
more often than did their control group counterparts. The program’s theory is that completing 
the College Success course and visiting the Success Centers will enable students to perform 
better academically during the program, but also, the skills and study habits they gain through 
these experiences may translate into longer-term success. During the second program semester 
(spring 2007), the experiences of program and control group students once again differed, 
although not nearly as dramatically as during the first program semester. Once the program was 
over, students’ use of the Success Centers was about the same, regardless of their experiences 
with the Enhanced Opening Doors program. The next section turns to the impact of the program 
on various outcomes during the first and second program semesters, as well as during the first 
and second postprogram semesters. 


The Effects of the Enhanced Opening Doors Program 

The Enhanced Opening Doors program was designed to help students on probation improve 
their performance in college and move off probation. This section describes the effects of the 
program on various educational outcomes, including students’ continued enrollment (persis- 
tence), academic standing (that is, probation status), grade point average (GPA), and credit 
accumulation. The outcomes described in this chapter are measured at Chaffey College only, 
since during the first two years after students entered this study only a small percentage of 
students attended or transferred to other institutions. Chapter 4, which focuses on the long-term 
effects of the Enhanced Opening Doors program, examines the impact of the program at 
Chaffey College and other postsecondary institutions. Box 3.1 explains how to read the tables 
that are discussed below and in Chapter 4. 
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Box 3.1 

How to Read the Impact Tables in This Report 

Most tables in this report use the format illustrated in the abbreviated table below, which displays 
some hypothetical transcript data and educational outcomes for the program and control groups. 
The first row, for example, shows that, by the end of the first semester, program group students 
had earned an average of 4.5 total credits and control group students had earned an average of 3.0 
total credits. 

The “Difference” column in the table shows the differences observed between the two research 
groups’ outcomes — that is, the program’s estimated impacts on the outcomes, or estimates of 
the true impacts (which are impossible to determine). For example, the estimated impact on 
average total credits earned can be calculated by subtracting 3.0 from 4.5, yielding an estimated 
impact of 1.5 average total credits earned. This difference represents the estimated impact of the 
program rather than the true impact because, although study participants are randomly assigned to 
the program and control groups, the result would have been different if a different group of 
students had been included in the study or if they had been randomized in a different way. 

Differences marked with one asterisk or more are statistically significant, meaning that there is a 
high probability that the program had an impact (positive or negative) on student outcomes. The 
number of asterisks indicates the probability that an impact at least as large as the one observed in 
the study would have occurred even if the program’s true impact had been zero. One asterisk 
corresponds to a 10 percent probability; two asterisks, a 5 percent probability; and three asterisks, 
a 1 percent probability. The more asterisks, the more likely the program had an impact on student 
outcomes. The impacts in the table excerpt below have three asterisks each, which indicates that 
they are statistically significant at the 1 percent level — meaning that there is only a 1 percent 
chance of observing an impact this large (or larger) if the program actually had no effect on total 
credits earned. In other words, there is a 99 percent level of confidence that the program had a 
positive impact on the average number of total credits earned. 

Also shown in the table is the standard error of the impact estimate. The standard error is a meas- 
ure of uncertainty or variability around the impact estimate. Some useful mles of thumb are that 
there is about a 90 percent chance that the true impact is within plus or minus 1.65 standard errors 
of the estimated impact, roughly a 95 percent chance that the true impact is within plus or minus 
1.96 standard errors of the estimated impact, and about a 99 percent chance that the true impact is 
within plus or minus 2.58 standard errors of the estimated impact. For example, in the first row of 
data below, there is roughly a 99 percent chance that the program’s impact on students’ average 
total credits earned lies between 0.5 and 2.5, calculated as 1.5 ± (2.58 x 0.4). 



Program 

Control 

Difference 

Standard 

Outcome 

Group 

Group 

(Impact) 

Error 

Cumulative total credits earned 





First semester 

4.5 

3.0 

\ 5 *** 

0.4 

Second semester 

9.1 

6.1 

3 Q *** 

0.8 
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Recall that the Enhanced Opening Doors program at Chaffey includes a three-credit 
College Success course (which was also offered to the control group, but not required, and 
which very few of them took — suggesting that in the absence of the program, students would 
be unlikely to take this course). The College Success course and its associated three credits 
cannot be applied toward a degree and cannot be transferred to another institution. As a result, 
for certain outcome measures (GPA and credit accumulation), results are presented both 
including and excluding this course. S * * 8 Interpreting the program’s short-term effects is complicat- 
ed, since taking the College Success course does not directly push students closer to a degree. 
However, the College Success course (and Success Center visits) may increase students’ 
chances of completing their other required courses and may improve their likelihood of passing 
courses in future semesters. 

Table 3.2 presents information on sample members’ academic performance during the 
two program semesters (fall 2006 and spring 2007) and the two subsequent postprogram 
semesters (fall 2007 and spring 2008). 

Persistence 

In order for students to graduate they must accumulate credits, which can occur only if 
they enroll in class. For this reason, any program that successfully increases enrollment rates has 
a very good chance of exerting an impact on students' credit accumulation and therefore their 
likelihood of graduating. Table 3.2 considers two measures of persistence — that is, continued 
enrollment — that are useful for interpreting the program’s impacts on other key short-term 
outcomes like probation status, grade point average, and credit accumulation. The first panel 
examines enrollment (registration) rates for program and control group members during the first 
through fourth semesters after students joined the study. These numbers help answer the 
question, “Did the Enhanced Opening Doors program have an impact on enrollment during 
semester x?” The second panel considers the average cumulative number of semesters that 
program and control group members enrolled since the start of the study. These numbers help 
answer the question, “Since the start of the study, did the Enhanced Opening Doors program 
have an impact on cumulative semesters enrolled as of semester x?” 


S GPA is calculated in two ways: (1) using grades earned in all courses except for developmental courses 

(this measure includes the College Success course), and (2) using grades earned only in degree-applicable 

courses, which excludes the College Success course, developmental courses, and other non-degree-applicable 
courses. Credit accumulation is also calculated in two ways: (1) including all credits, and (2) including degree- 

applicable credits where students earned a “C” or higher and developmental credits. (That is, this excludes the 
College Success course as well as a nominal number of other non-degree-applicable courses that are not 
developmental courses.) 
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The Opening Doors Demonstration 
Table 3.2 

Academic Outcomes Among Sample Members at Chaffey College, 
Enhanced Opening Doors Study, First Four Semesters 


Chaffey College Four-Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Registered for any courses (%) 

First semester 

85.3 

80.9 

4.4 

3.6 

Second semester 

67.9 

61.8 

6.1 

4.5 

Third semester 

41.5 

47.3 

-5.8 

4.7 

Fourth semester 

36.6 

36.8 

- 0.2 

4.6 

Cumulative number of semesters enrolled 

First semester 

0.85 

0.81 

0.04 

0.04 

Second semester 

1.53 

1.43 

0.10 

0.07 

Third semester 

1.95 

1.90 

0.05 

0.10 

Fourth semester 

2.31 

2.27 

0.04 

0.13 

In good academic standing 2 (%) 

First semester 

21.9 

12.3 

9.6 *** 

3.6 

Second semester 

24.1 

13.6 

10.5 *** 

3.7 

Third semester 

20.1 

16.4 

3.7 

3.7 

Fourth semester 

20.1 

19.1 

1.0 

3.8 

Cumulative GPA higher than 2.0 b (%) 

First semester 

40.2 

21.8 

j g 4 *** 

4.3 

Second semester 

36.6 

25.5 

11.2 ** 

4.4 

Third semester 

36.2 

30.0 

6.2 

4.5 

Fourth semester 

34.8 

30.5 

4.4 

4.5 

Cumulative degree-applicable GPA 
higher than 2 . 0 C (%) 

First semester 

24.1 

21.4 

2.7 

4.0 

Second semester 

30.8 

25.0 

5.8 

4.3 

Third semester 

29.9 

30.0 

- 0.1 

4.4 

Fourth semester 

29.9 

30.5 

-0.5 

4.4 

Cumulative credits earned 

First semester 

4.6 

3.2 

j 4 *** 

0.4 

Second semester 

9.3 

6.4 

3 Q *** 

0.8 

Third semester 

11.6 

8.6 

3 j *** 

1.1 

Fourth semester 

14.0 

10.8 

3.2 ** 

1.4 

Cumulative degree-applicable and 
developmental credits earned 1 * 

First semester 

2.6 

2.7 

- 0.1 

0.3 

Second semester 

6.3 

5.4 

1.0 

0.7 

Third semester 

8.3 

7.3 

1.0 

0.9 

Fourth semester 

10.4 

9.3 

1.1 

1.2 

Sample size (n = 444) 

224 

220 




(continued) 
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Table 3.2 (continued) 


SOURCES: MDRC calculations from Chaffey College transcript and probation data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Cumulative measures are based on semesters 1 through 4; summer terms are included in spring semester 
results. 

a Sample members who were not enrolled were not considered to be in good academic standing. 

b Grades earned in all courses except for developmental courses are used in the calculation of GPA. Students 
who do not have a GPA for this time period are included as zeros. 

c Grades earned only in degree -applicable courses, which exclude the College Success course, are used in this 
calculation of GPA. Students who were not registered and thus do not have a GPA for this time period are 
included as zeros. 

includes credits earned in courses taken during the first four semesters of the study and excludes credits from 
the College Success course. If a student receives a letter grade in a course, this measure only includes credits in 
which the student earned a “C” or higher. If a student takes a class “pass/fail” or if the class is only offered 
“pass/fail,” then this measure only includes a grade of “pass.” 


During the first program semester (fall 2006), 85.3 percent of program group students 
and 80.9 percent of control group students registered for at least one course. This difference is 
not statistically significant. It is not surprising that the differences in registration rates during the 
first program semester were not significant, because registration occurred before the program 
group students received many program services, if any. During the second program semester 
(spring 2007), the program had an estimated impact of 6.1 percentage points on students’ 
likelihood of enrolling at Chaffey College. Although this impact is not statistically significant, its 
magnitude may be considered significant in practical terms, despite a great deal of uncertainty 
around its accuracy. 9 Given this study’s limited sample size of 444, the impact would have to 
have been approximately 7.5 percentage points in order to be deemed statistically significant. 
While the second program semester impacts are somewhat promising with respect to persistence, 
in the subsequent two postprogram semesters (fall 2007 and spring 2008), the program did not 
have meaningful effects on current enrollment or on cumulative number of semesters enrolled. 

Good Academic Standing 

A key goal of Chaffey’ s Enhanced Opening Doors program is to move students off 
probation and into good academic standing. Achieving this goal may get students back on track 
toward accumulating the necessary credits to earn a certificate or degree, or to transfer to a four- 


9 This study’s sample size is only 444 students — much smaller than the typical sample size for an MDRC 
study of a program designed to help community college students. (Usual sample sizes exceed 1,000 students.) 
As a result, the smallest “true” impact that can be detected with a high degree of confidence is fairly large. 
Consequendy, while the 6.1 percentage point impact has a large margin of error, it may still be noteworthy. 
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year college. The third panel of Table 3.2 displays the percentage of students in good academic 
standing during each semester. Only registered students are considered eligible to be counted 
among those in good academic standing. 10 

During the first program semester, 22 percent of program group members were in good 
academic standing, whereas 12 percent of control group members were in good academic 
standing. The estimated 10 percentage point difference is statistically significant, showing that 
the program improved students’ chances of being in good academic standing. In the second 
program semester, 24 percent of program group members were in good academic standing, 
whereas 14 percent of control group members were in good academic standing. The estimated 
impact of the program is 10 percentage points, about the same as it was during the first semes- 
ter. In line with the program's theory, during the two program semesters, students were able to 
get off probation at a higher rate than they would have without the program. However, after the 
program was complete, during the following two postprogram semesters, program and control 
group students were in good academic standing at similar rates, suggesting that this program 
effect was short-lived. 

Grade Point Average 

Grade point average is a common indicator of academic performance. Although some 
researchers believe that GPA provides little information about what students have actually 
learned, 11 it may still represent an amalgamation of factors like learning, effort, and tenacity. 
The fourth and fifth panels of Table 3.2 show two different versions of cumulative GPA over 
four semesters. The first version uses the student's cumulative GPA for all credit-bearing 
courses taken since the start of the study, whether or not they can be applied toward a degree. 
The second version uses the student’ s cumulative GPA only for degree-applicable courses taken 
since the start of the study — that is, this second measure excludes credit from the College 
Success course and other courses that are not applicable toward a degree. Shown under both 
GPA definitions are the percentages of students with a GPA higher than 2.0. 12 

During the first program semester, more program group students than control group 
members (by 18 percentage points) had a GPA higher than 2.0, when the College Success course 
is included in the GPA calculation. By the end of the second semester of the study, the program’s 
impact on GPA remained positive and significant, although the effect appears to have been 
diminishing. Two years after the study began (fourth semester), there was a difference of only 4 


10 In other words, students who do not register are automatically considered “not in good academic standing.” 
n Adelman (2004). 

12 Students who have not registered since random assignment are considered not to have a GPA of 2.0 or 
higher. 
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percentage points between program group members and control group members with a cumula- 
tive GPA higher than 2.0 — continuing the trend of a diminishing program effect on GPA. 

MDRC’s earlier report on the Enhanced Opening Doors program (as well as Table 1.2 
in this report) showed that at the end of the first two program semesters the program had a 
statistically significant impact on cumulative degree-applicable GPA. 13 This finding was viewed 
as particularly promising since it is not an artifact of the required non-degree-applicable College 
Success course. Unlike the original analyses, in Table 3.2 the second program semester cumula- 
tive degree-applicable GPA includes the summer term (data that were not available at the time 
of the earlier report). As can be seen, by the end of the second program semester (summer 
included), the program’s estimated impact on cumulative degree-applicable GPA is still 
positive, but is not statistically significant. Continued follow-up through two postprogram 
semesters shows that the percentage of program and control group members who have a 
cumulative degree-applicable GPA of 2.0 or higher is very similar during the semesters after the 
program is complete. 

Credit Accumulation 

Credit accumulation is a key indicator that students are making progress toward a de- 
gree. In order for Chaffey students to earn an associate’s degree, they must earn at least 60 
degree- applicable credits. 14 For those students who place into developmental coursework, 
developmental credits earned (which are not applicable toward a degree) may also indicate 
progress toward a degree since students are encouraged to take these courses before moving on 
to degree-applicable courses. 15 In addition to degree-applicable credits and developmental 
credits, Chaffey College offers courses that are not developmental and not degree-applicable, 
like the College Success course. The last two panels in Table 3.2 exhibit two measures of credit 
accumulation: (1) cumulative credits earned, which includes all forms of credits earned (includ- 
ing the three credits offered by the College Success course, which are not applicable toward a 
degree) and (2) cumulative degree-applicable and developmental credits earned, which are the 
types of credits that are most relevant to earning a degree. 


l3 Scrivener, Sommo, and Collado (2009). 

l4 Courses with degree-applicable credits do not include developmental courses, the three-credit College 
Success course that is a requirement of the Enhanced Opening Doors program at Chaffey College, or a nominal 
number of other courses. 

I5 A 1988 lawsuit in California brought forward by the Mexican American Legal Defense Fund challenged 
educational policies that disproportionately directed Latinos into developmental courses. As part of the 
settlement, the California Community College Board of Governors proposed a complex set of revisions to Title 
5 regulations regarding prerequisites, assessment, and student placement. These regulations and their import for 
local practice is still a matter of debate; however, they have been widely interpreted by community college 
administrators to prohibit mandatory placement (Perry, Bahr, Rosin, and Woodward, 2010). 
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With respect to cumulative credits earned (shown in the second-to-last panel of Table 
3.2), over the course of the first four semesters after random assignment, the program had a 
strong positive effect. The estimated impact during the first through fourth semesters was 1 .4 
credits, 3.0 credits, 3.1 credits, and 3.2 credits, respectively. Since these numbers are cumula- 
tive, past program effects carry forward to the future. In this case, it appears that the majority of 
the impact on cumulative credits earned occurred during the two program semesters, and the 
effect was maintained during the two postprogram semesters. 

A comparison of cumulative degree-applicable and developmental credits earned (last 
panel in Table 3.2) with all cumulative credits earned (second-to-last panel) indicates how much 
of the impact on total credits earned is a result of the non-degree-applicable College Success 
course. 16 Notably, the first program semester impact on credit accumulation is driven entirely by 
non-degree-applicable credits, as demonstrated by the estimated impact of near zero on degree- 
applicable and developmental credits earned. Since this measure ignores the three-credit College 
Success course, which most program group students took during the first semester, the fact that 
program group students did not fall behind control group members on degree-applicable and 
developmental credits is consistent with the program developers’ expectations for this time 
period. In other words, despite having an additional three-credit class as part of their course load, 
program group students managed to keep pace with their control group counterparts. The skills 
and study habits learned in the College Success course may translate into better performance in 
future courses, resulting in additional credits earned during the second semester and beyond, after 
most program group students completed a College Success course. 

Looking at degree-applicable and developmental credits earned in the second, third, and 
fourth semesters, however, fails to confirm that hypothesis: the program’s positive effects on the 
total credit accumulation measure do not appear to have translated into meaningfully large 
positive impacts on cumulative degree-applicable and developmental credits earned. 


Summary 

Overall, the Enhanced Opening Doors program at Chaffey College had positive impacts on 
students’ academic standing, GPA, and total credit accumulation (including credits awarded for 
the College Success course) through two semesters after random assignment. However, there is 
no evidence of meaningful impacts on enrollment rates or accumulation of degree-applicable 
and developmental credits. The next chapter follows the same cohort of students for up to four 
years after the MDRC Enhanced Opening Doors study began, to assess whether the program 
has positive impacts on students’ long-term academic outcomes. 


I6 A nominal number of other non-degree- applicable courses are also excluded from this measure. 
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Chapter 4 

Long-Term Impacts 


Chapter 3 of this report — which focused on the effect of the Enhanced Opening Doors pro- 
gram on students’ outcomes during the first two years of the study — showed that the oppor- 
tunity to participate in the program improved some student outcomes during the two program 
semesters when the program was running, although the effects largely faded by the end of the 
first two postprogram semesters, after the program had been completed. This chapter presents 
new findings related to the program’s impact on students’ academic progress and completion, as 
defined below, during the academic year in which the program ran, 1 as well as in the three 
subsequent postprogram years, for a total of four years of follow-up after the study began. 
Figure 4.1 maps the time period described in this chapter, with black arrows representing the 
program semesters. 

This long-term follow-up report focuses on whether the program helped improve stu- 
dents’ academic progress and/or completion, as measured by three indicators of success: 

1 . Persistence, or continued enrollment — indicates progress 

2. Credit accumulation — indicates progress 

3. Attainment of a degree or certificate — indicates completion 

Summary of Key Findings 

Despite the program's positive effects during the two semesters in which it ran, long-term 
academic progress and completion among students in Enhanced Opening Doors was not 
significantly different from the long-term academic progress and completion among students in 
the control group, who could receive Chaffey College’s usual services (see Table 4.1). A 
summary of the key findings follows: 

• Persistence. Four years after students entered the study, those who were ran- 
domly assigned to the Enhanced Opening Doors program had enrolled in 
college for almost the exact same number of semesters as their control group 
counterparts (4.2 compared with 4.1, respectively). The estimated 0.1 differ- 
ence is not statistically significant. 

'This time period includes the two program semesters — fall 2006 and spring 2007 — as well as the 
summer 2007 semester. That is, data were collected for fall, spring, and summer, but the program ran during 
the fall and spring semesters only. 
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The Opening Doors Demonstration 

Figure 4.1 

Data Collection Timeline, Years One Through Four of Enhanced Opening Doors Study 
Chaffey College Four- Year Follow-Up Report 



Year 1 


Year 2 


Year 3 


Year 4 


Program semester 


| — | Postprogram semester 


NOTES: Enrollment and credits earned are based on data from fall 2006 through summer 2010. Degree and certificate data are from fall 2006 
through spring 2010. 



The Opening Doors Demonstration 
Table 4.1 


Primary Academic Outcomes Among Sample Members, 
Enhanced Opening Doors Study, Four Years After Random Assignment 

Chaff ey College Four-Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Number of semesters enrolled at any college 

4.18 

4.06 

0.12 

0.24 

Cumulative degree-applicable and developmental 

credits earned 3. 

18.4 

17.1 

1.4 

1.9 

Enrolled at any college in year 4 or earned 

a degree or certificate (%) 

45.5 

45.5 

0.1 

4.7 

Enrolled at any college in year 4 

44.2 

44.1 

0.1 

4.7 

Earned a degree or certificate 

7.6 

6.4 

1.2 

2.4 

Sample size (n = 444) 

224 

220 




SOURCES: MDRC calculations from Chaffey College transcript, California Community College Chancellor's 
Office, and National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Enrollment and credits earned are based on data from fall 2006 through summer 2010. Degree and certificate 
data are from fall 2006 through spring 2010. 

“Includes credits earned in courses taken during the first four years of the study and excludes credits from the 
College Success course. If a student receives a letter grade in a course, this measure only includes credits in 
which the student earned a “C” or higher. If a student takes a class “pass/fail” or if the class is only offered 
“pass/fail,” then this measure only includes a grade of “pass.” 
b At any public, two-year California college. 


• Credit accumulation. Four years after students entered the study, those who 
were randomly assigned to the Enhanced Opening Doors program had not 
earned significantly more credits (combining degree-applicable and devel- 
opmental credits, but excluding the College Success course credits) than their 
control group counterparts. 

• Attainment of a degree or certificate. Four years after students entered the 
study, those who were randomly assigned to the Enhanced Opening Doors 
program had not earned a degree or certificate at a significantly higher rate 
than their control group counterparts. 

Greater details on these findings are presented throughout the remainder of this chapter. 
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Detailed Findings 

Measuring the long-term effects of a program like Enhanced Opening Doors at Chaffey College 
can be complicated by several factors. First, the goals of community college students vary. 
Some seek to earn a certificate, some seek to earn an associate’ s degree, and still others plan to 
transfer to a four-year institution to complete the requirements for a bachelor’s degree and 
beyond. Second, data limitations mean that not all indicators of academic progress or success 
can be captured at all institutions. Third, the goals of students may not always align perfectly 
with the goals of the college; for instance, while the college is invested in seeing its students 
graduate there, the students may not care whether they graduate from that college or another one 
as long as they get a degree. Despite these challenges, examining three main indicators of 
academic progress and completion, measured at various institutions (that is, at Chaffey College 
only, at California community colleges only, and at all colleges, both two- and four-year, 
throughout the United States), 2 can provide a thorough picture of the long-term success of the 
Enhanced Opening Doors program. 

Notably, from the perspective of the student, academic progress or completion at 
Chaffey College or at any other college may be viewed as a positive outcome; however, it is 
also interesting to note whether students succeeded at Chaffey College alone, since that is where 
students began this study. MDRC’s primary focus is on the student perspective, so the main 
findings in this chapter are presented for all institutions, but Appendix Table A.5 includes 
results for Chaffey College alone. 

The detailed findings presented below are organized by the three indicators of college 
success — persistence, credit accumulation, and attainment of a degree or certificate — with 
the measures described first, followed by a description of the program’s impacts. 

Persistence 

Measures 

Since students can only earn credits if they enroll in college, continued enrollment is a 
good proxy measure for students’ progress toward a degree, especially when information on 
credit accumulation is unavailable. 3 Table 4.2 depicts continued enrollment during the first four 
years after the study began. The first panel focuses on the cumulative number of semesters in 
which students had been enrolled as of the end of the first, second, third, and fourth years after 
the study began. For example, at the end of the program year, program group students had been 

2 Subject to the limitations of the National Student Clearinghouse data. 

3 Given this study’s data sources, credit accumulation data are unavailable for four-year colleges and pri- 
vate colleges in California, and all colleges outside of California. 
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The Opening Doors Demonstration 

Table 4.2 


Enrollment at Any U.S. College Among Sample Members, 
Enhanced Opening Doors Study, Years One Through Four 


Chaffey College Four-Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Cumulative number of semesters enrolled 

Year 1 

1.59 

1.52 

0.07 

0.06 

Year 2 

2.63 

2.56 

0.07 

0.13 

Year 3 

3.48 

3.35 

0.12 

0.19 

Year 4 

4.18 

4.06 

0.12 

0.24 

Number of colleges ever attended 

1.41 

1.44 

-0.03 

0.08 

Enrolled at any U.S. college (%) 

Year 1 

90.6 

90.9 

-0.3 

2.8 

Year 2 

62.0 

62.7 

-0.7 

4.6 

Year 3 

51.8 

45.9 

5.9 

4.7 

Year 4 

44.2 

44.1 

0.1 

4.7 

Enrolled at any 4-year U.S. college (%) 

Year 1 

1.3 

3.2 

-1.8 

1.4 

Year 2 

4.5 

3.6 

0.8 

1.9 

Year 3 

6.2 

6.8 

-0.6 

2.4 

Year 4 

8.0 

7.7 

0.3 

2.6 

Enrolled at any 2-year U.S. college (%) 

Year 1 

90.2 

90.0 

0.2 

2.8 

Year 2 

58.5 

61.8 

-3.4 

4.7 

Year 3 

47.8 

41.8 

5.9 

4.7 

Year 4 

37.1 

38.2 

-1.1 

4.6 

Sample size (n = 444) 

224 

220 




SOURCES: MDRC calculations from Chaffey College transcript, California Community College Chancellor's 
Office, and National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 
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enrolled for an average of 1.59 semesters, whereas control group students had been enrolled for 
an average of 1.52 semesters. (The difference is not statistically significant). The remaining 
panels focus on the percentage of students currently enrolled during each of the four follow-up 
years (that is, not cumulatively.) 4 For example, 90.6 percent of program group members and 
90.9 percent of control group members enrolled at any U.S. college at some point during the 
year after the study began. Similar enrollment data are provided for any four-year college and 
any two-year college. 

Over the long term, continued enrollment can be somewhat difficult to interpret be- 
cause some students stop enrolling after they have earned a degree or certificate. Thus, over 
the long term, lack of enrollment can be a positive outcome. As a result, the outcome “en- 
rolled or earned a degree/certificate” is also examined later in this chapter to determine the 
program’ s long-term effects. 

Impacts 

Before discussing the program’s impact on persistence (by comparing program group 
students’ outcomes with those of the control group students), it is worth noting the overall 
enrollment trends through the first four years of the study, as shown in Table 4.2. Mimicking 
national trends, many students in this study “stopped out” (that is, did not enroll in any classes 
for a period of time and then returned to school) or dropped out over the four-year follow-up 
period. During the first year, just over 90 percent of all students in the study sample enrolled at 
any college. Enrollment rates declined sharply in the second year, when just over 60 percent of 
all students enrolled at any college — nearly a 30 percentage point drop-off from the previous 
year. The sharp decline leveled off a bit in the third and fourth years, but nonetheless, enroll- 
ment rates in the fourth year hovered around 44 percent. As explained in more detail later in this 
chapter, the decline was not attributable to large numbers of students earning degrees or 
certificates. (Only about 7 percent achieved this goal by the end of the fourth year.) These 
overall trends reaffirm the general challenge of getting students to persist and complete the 
requirements for a degree in community college. 

Given that challenge, one interest of this study was whether Enhanced Opening Doors 
led program group students to enroll in college at a higher rate. Throughout the four-year 
follow-up period, students in the program group did not enroll in college at a significantly 
higher rate than their control group counterparts. The results are consistent whether persistence 
is measured cumulatively (that is, comparing the total number of semesters in which students 
enrolled) or based on students’ current enrollment status (that is, comparing the percentage of 


4 Students are counted as enrolled in a given year if they enrolled for a class in at least one semester during 
that year. 
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students enrolled during a given year). Table 4.2 also shows that enrollment rates in four-year 
colleges and two-year colleges were roughly the same for the program and control groups — 
that is, no statistically significant differences were found between the two groups, meaning that 
there is no strong evidence that the program helped (or harmed) students compared with the 
college’ s usual services. 

Overall, with respect to the likelihood that students will continue to enroll in college, 
there is little evidence that the program had added value above and beyond the college’s usual 
services. 

Credit Accumulation 

Measures 

As noted in Chapter 3, credit accumulation is a key indicator that students are making 
progress toward a degree. Some of the impacts of Enhanced Opening Doors during the program 
semesters were driven partially by the non-degree-applicable credits that students earned in the 
program’s three-credit College Success course; consequently, it is important to assess whether 
these findings translate into longer-term impacts on measures that focus on the courses students 
need to take in order to graduate. For this reason, credit accumulation is examined in two ways. 
The first measure includes all credits, and the second measure includes degree-applicable and 
developmental credits only. 5 

During the program year, it would have been challenging for program group students to 
earn more degree-applicable credits plus developmental credits than their control group coun- 
terparts, since their course loads included the College Success course, which most of the control 
group members did not take. However, during the three postprogram years, differences in 
credits earned in degree-applicable courses plus developmental courses (that is, the courses 
students need to complete in order to graduate) would indicate whether the program helped 
students progress toward a degree. 

Impacts 

Figure 4.2 plots the program’s impact on cumulative total credits earned (including the 
College Success course) during each of the four years after the study began. During the program 


5 Data on credit accumulation are limited to two-year public colleges in the state of California. As a result, 
if program and control group members enroll in and/or earn credits at a different rate at the institutions where 
data are not available, then the estimated impact on credit accumulation could be biased. However, program 
and control group students enrolled in the colleges where credit accumulation data are unavailable at similar 
rates; consequently, credit accumulation data are unlikely to be significantly biased. 
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The Opening Doors Demonstration 

Figure 4.2 


Cumulative Credits Earned Among Sample Members at Any Public, Two-Year 
California College, Enhanced Opening Doors Study, Years One Through Four 
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Chaffey College Four-Year Follow-Up Report 



■ Program group □ Control group 

SOURCES: MDRC calculations from Chaffey College transcript and California Community College 
Chancellor's Office data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 
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year, program group students earned 9.4 total credits, whereas their control group counterparts 
earned 6.9 total credits, for an estimated program impact of 2.5 credits (nearly one full course). 
This positive impact, which is statistically significant, persisted through the end of the four-year 
follow-up period, at which point program group students had earned 3.6 more credits than their 
control group counterparts, most of which was carryover from the first-year effect. While the 
impact on total credits earned is driven in part by the College Success course credits, the skills 
and strategies learned in this course could have translated into long-term impacts on credit 
accumulation in other classes that counted toward a degree. 

Figure 4.3 shows the program’s impact on degree-applicable plus developmental credits 
earned. Despite the program’ s promising effects on total credit accumulation (that is, including 
the non-degree-applicable College Success course), these effects do not translate into significant 
impacts when the College Success course credits are excluded. During each of the four years of 
follow-up, the average cumulative number of degree-applicable plus developmental credits 
earned was similar for the program and control groups. At the end of the first year (the program 
year), students in the full study sample (program and control groups) earned an average of 
around 6 credits; by the end of the fourth year, students had earned an average of around 17 to 
18 credits, with no practically meaningful (or statistically significant) differences between 
program and control groups along the way. 

Appendix Table A.2 provides additional information on credit accumulation. 

Attainment of a Degree or Certificate 

Measures 

In the absence of strong evidence that Enhanced Opening Doors significantly increased 
credit accumulation, it is unlikely that the program increased students’ chances of earning a 
degree or certificate. However, it is theoretically possible that some of the “college knowledge” 
that students acquired in the College Success course could have helped them focus on taking the 
“right” courses to put them on a path toward a degree. In other words, although students in the 
program did not earn more credits than their control group counterparts, if the credits they did 
earn were more directly associated with earning a degree, the program still may have had an 
impact on the attainment of a degree or certificate. This section examines degree or certificate 
attainment during the four years after the study began. 6 


6 For CCCCO schools, only certificates that require 18 credits or more are included. 
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The Opening Doors Demonstration 
Figure 4.3 

Cumulative Degree-Applicable and Developmental Credits Earned Among 
Sample Members at Any Public, Two-Year California College, 
Enhanced Opening Doors Study, Years One Through Four 

Chaffey College Four-Year Follow-Up Report 



■ Program group □ Control group 


SOURCES: MDRC calculations from Chaffey College transcript and California Community College 
Chancellor's Office data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. No statistically significant 
differences between research groups were observed. 

Estimates are adjusted by round of random assignment. 

"Cumulative degree-applicable and developmental credits earned" includes credits earned in courses 
taken during the first four years of the study and excludes credits from the College Success course. It includes 
only degree-applicable courses in which students received a grade of “C” or higher, or, for pass/fail courses, 
which include developmental courses, a passing grade. 
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In addition to degree/certificate attainment alone (which reflects “completion”), a 
measure that combines degree/certificate attainment with enrollment (which reflects “pro- 
gress”) is also examined, to ensure that the program’s impacts on both progress and comple- 
tion are captured. Many community college students aspire to earn a four-year degree, and 
among them, some transfer to a four-year institution before earning a degree or certificate at 
their community college; for those students, continued enrollment may be a sign of continued 
success. Thus, this measure counts anyone who is either still enrolled or has completed a 
degree or certificate as a success. 

Before examining the results, recall that the target population in this study comprises 
students on probation, a high-risk group of students with a low likelihood of earning a creden- 
tial. However, on average, students in the research sample had earned around 14 credits prior to 
the start of the study, so they had made some progress toward earning a degree when they 
entered the program. With that in mind, presented below are the program’s effects on de- 
gree/certificate attainment. 

Impacts 

Figure 4.4 displays the percentage of students who earned a degree or certificate at any 
school by the end of each year of follow-up. What is most striking about the degree/certificate 
attainment measure is not whether the program had a significant impact on attainment, but 
rather the low rate of attainment among all study participants. Among the full study sample 
(program and control), only around 7 percent of all students had earned a degree or certificate 
four years after the study began. Not surprisingly, given the results for persistence and credit 
accumulation, the program did not have a significant impact on degree/certificate attainment. 

While the rates of degree/certificate attainment presented in Figure 4.4 are low, consid- 
er the percentage of students who were still enrolled in college or had earned a degree at the end 
of the follow-up period — an indication that these students are still progressing toward gradua- 
tion or have completed an academic goal, as opposed to students who have dropped out com- 
pletely. As shown in Figure 4.5, four years after the study began, 46 percent of all students in 
this study were either still enrolled in some college courses or they had earned a degree or 
certificate. In other words, at least some of the students in the research sample may still be 
progressing, despite the low rates of completion. However, there is not meaningful evidence 
that the Enhanced Opening Doors program helped or impeded this long-term progress toward or 
completion of the requirements for a degree. 

Appendix Table A.3 provides more detailed information on the same measures present- 
ed in Figures 4.4 and 4.5. 
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The Opening Doors Demonstration 
Figure 4.4 

Percentage of Students in the Study Sample Who Ever Earned a Degree or Certificate 
at Any U.S. College, Enhanced Opening Doors Study, Years One Through Four 


Chaffey College Four-Year Follow-Up Report 



Year 1 Year 2 Year 3 Year 4 


■ Program group □ Control group 


SOURCES: MDRC calculations from Chaffey College transcript, California Community College Chancellor's 
Office, and National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. No statistically significant 
differences between research groups were observed. 

Estimates are adjusted by round of random assignment. 

Degree and certificate data are from fall 2006 through spring 2010. 

Percentages who earned a degree or certificate are shown directly above each bar. Percentage point 
differences between the program and control groups are shown above the percentages who earned a degree or 
certificate. 
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The Opening Doors Demonstration 

Figure 4.5 

Currently Enrolled or Earned a Degree or Certificate Among Sample Members at 
Any U.S. College, Enhanced Opening Doors Study, Years One Through Four 

Chaffey College Four-Year Follow-Up Report 
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■ Program group □ Control group 


SOURCES: MDRC calculations from Chaffey College transcript, California Community College Chancellor's 
Office, and National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. No statistically significant 
differences between research groups were observed. 

Estimates are adjusted by round of random assignment. 

Enrollment is based on data from fail 2006 through summer 2010. Degree and certificate data are from fall 
2006 through spring 2010. 

Percentages who earned a degree or certificate are shown inside the bars. Percentage point differences 
between the program and control groups are shown above the bars. 
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Whereas Figures 4.4 and 4.5 focus on earning a degree or certificate, Table 4.3 breaks 
these results out by type of degree. There is some evidence suggesting that program group 
students earned a bachelor’s degree at a higher rate than their control group counterparts (1.8 
percent compared with 0.0 percent); however, this result should be viewed with great caution 
since the 1.8 percentage point difference represents only four students. 7 While over 90 percent 
of the students in this study did not earn any degree or certificate by the end of the four-year 
follow-up period, the associate’s degree was the most common accomplishment among those 
who earned a degree or certificate. 


The Opening Doors Demonstration 
Table 4.3 

Highest Degree or Certificate Earned at Any U.S. College Among Sample Members, 
Enhanced Opening Doors Study, Four Years After Random Assignment 


Chaffey College Four-Year Follow-Up Report 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Bachelor's degree 

1.8 

0.0 

1.8 ** 

0.9 

Associate's degree 

4.0 

5.5 

-1.5 

2.0 

Certificate 

1.3 

0.9 

0.4 

1.0 

Unknown degree or certificate type 

0.4 

0.0 

0.4 

0.5 

No degree or certificate 

92.4 

93.6 

-1.2 

2.4 

Sample size (n = 444) 

224 

220 




SOURCES: MDRC calculations using Chaffey College transcript, California Community College Chancellor's 
Office, and National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Degree and certificate data are from fall 2006 through spring 2010. 


7 In addition, caution is urged because this outcome is considered “exploratory” or “secondary.” The “con- 
firmatory” or “primary" analyses described in Table 4.1 “assess how strongly the study’s pre-specified central 
hypotheses are supported by the data.” In contrast, the “exploratory” or “secondary” analyses shown in Table 
4.3 “identify hypotheses that could be subject to future rigorous testing.” See Schochet (2008). 
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Effects for Selected Subgroups 

In addition to the analyses described above, Appendix Table A.4 presents the primary 
outcomes for sample members who had been on probation for one semester when they entered 
the study and sample members who had been on probation for two or more semesters at that 
point. Findings presented in the initial MDRC report on the program suggested that the program 
was more effective for students who were in their first semester of probation. 8 However, given 
the already small sample size in the main analyses, this subgroup analysis is underpowered — 
that is, the number of students in each group is small enough that the program would have to be 
substantially more effective for one group compared with the other to observe statistically 
significant differences in impacts. Not surprisingly, the program’s effects on students’ long-term 
outcomes did not differ significantly based on their length of time on probation. 

Chaffey College Only 

Appendix Table A.5 presents academic outcomes (persistence, credit accumulation, and 
degree/certificate attainment) at Chaffey College only. The results are generally similar to those 
found when data from other colleges are included, with one exception: there is some evidence 
that by the end of the third year, students in the program group earned a degree or certificate at 
Chaffey College at a higher rate than their control group counterparts. 


Summary 

Overall, there is not strong evidence that the Enhanced Opening Doors program was particular- 
ly helpful (or harmful) to students’ long-term academic progress or completion, when compared 
with Chaffey College’s usual services. These results reinforce the notion that short-term 
interventions may have short-term impacts, but it is far more difficult to find short-term inter- 
ventions that have long-term impacts. 


8 Scrivener, Sommo, and Collado (2009). 
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Chapter 5 

Cost-Effectiveness of Enhanced Opening Doors 


This chapter is designed to answer questions regarding the decision to invest in College Success 
courses for probationary students at community colleges. For example, what is the program 
cost? What is the cost of the college’ s usual services for probationary students in the absence of 
a targeted program? What is the net cost of the program compared with the usual services? How 
can the net cost be lowered? What do net program costs look like when compared with the 
program outcomes? Finally, what are the policy implications of the program’s relative cost- 
effectiveness? 


Key Findings 

• During the two program semesters, each program group member represented 
a cost of approximately $4,300, and each control group member represented 
a cost of approximately $3,000. As a result, the net cost of the Chaffey Col- 
lege Enhanced Opening Doors program during the two program semesters 
was about $1,300 per program group member. 

• The cost per credit earned for the program group was slightly lower ($516) 
than the cost per credit earned for the control group ($530) during the two 
program semesters. The relative cost-effectiveness of earning credits in the 
short term fades away when only degree-applicable and developmental 
credits are considered — that is, when the College Success course credits 
are excluded. 

• Over the long run, impacts on key outcome measures are not great enough to 
bring about substantial changes in the cost of achieving desired outcomes. 


Methodology 

This chapter estimates the cost-effectiveness of the Enhanced Opening Doors program at 
Chaffey College that took place during the fall 2006 and spring 2007 semesters. The 
relationship between program costs and program impacts is examined in the short -term, which 
includes the two program semesters, and over the long term, which includes four years of 
follow-up. Short-term costs are considered from the perspective of Chaffey College, while long- 
term costs are considered from the perspective of California two-year public colleges as a whole 
(that is, sample members who attended or transferred to other schools were followed up); this 
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approach is intended to reflect the reality that Chaffey College students may attend or transfer to 
other schools in the long run. All program costs are expressed in dollars, but the monetary value 
associated with educational outcomes, such as moving into good academic standing and 
increased credit accumulation, is not estimated in this analysis. The analysis includes only costs 
that are part of the “steady state” of operation, which means that start-up and research costs 
have been excluded. 

The cost of program components is estimated using program budget documents from 
Chaffey College. The funding levels in the budget planned for higher rates of participation than 
actually occurred. For example, 17 sections of the College Success course were budgeted for 
the fall of 2006, but only 1 1 actually took place. Similarly, for the spring semester of 2007, only 
6 of 11 budgeted sections were filled. Spending levels during the program year were not 
adjusted based on this change. As a result, costs in this analysis should be considered high-end 
estimates because in the long run the college may be able to make adjustments that would 
decrease the actual cost per participant. This analysis does not capture adjustments that the 
college may have made beyond the program year. 

In addition to the cost of program components, this analysis estimates the cost of other 
college courses (that is, courses other than the College Success course) using transcript data as 
well as collegewide financial and enrollment data. For the short term, the cost of other college 
courses per sample member is estimated by multiplying the number of college credits attempted 
at Chaffey College per sample member by the Chaffey collegewide cost per credit. For the long 
term, the cost of other college courses per sample member is estimated by multiplying the 
number of college credits attempted at any California two-year public college by the Chaffey 
collegewide cost per credit. 

The cost per credit used to calculate the total cost of other college courses is estimated 
to be $234 for the 2006-2007 academic year. 1 It is estimated by dividing the Chaffey College 
operating budget (including fringe benefits) by the number of credits taken at Chaffey. 2 The 
number of credits taken at Chaffey is estimated by multiplying the number of full-time equiva- 
lent (FTE) students at Chaffey by 30, which corresponds to the California Community College 
Chancellor’s Office (CCCCO) definition of an FTE student. 3 The cost per credit for each 


'Once a cost per credit is estimated, the cost of college credits (other than the College Success credits) is 
assumed to be the same ($234 per attempted credit) for both the program and control group members. 

2 The cost of Enhanced Opening Doors is excluded from the operating budget just as Enhanced Opening 
Doors credits are excluded from the number of credits taken during that period. The operating budget is taken 
from an analysis of Chaffey College’s 2006-2007 finances by Economic Modeling Specialists Inc.; see 
www.chaffey.edu/research/lR_PDF_Files/Planning_Documents/EconomicContribution-MainReport.pdf 
(Robison and Christophersen, 2008). 

3 The number of FTE students at Chaffey is taken from the CCCCO Web site. See CCCCO (2011a, 201 lb). 
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follow-up year is estimated by adjusting the 2006-2007 academic year value by the Higher 
Education Price Index for two-year public colleges and by an annual discount rate of 3.5 
percent. 4 As a result, the cost per credit is $234 in the first year of follow-up, $233 in the second 
year of follow-up, $237 in the third year of follow-up, and $230 in the fourth year of follow-up. 
The Chaffey College cost per credit appears to be typical of California two-year public colleges, 
which is why this analysis applies the Chaffey College cost per credit to credits attempted at any 
California two-year public college. 


Analysis 

The sections that follow include more detailed discussion of the short-term and long-term costs 
of the Enhanced Opening Doors program at Chaffey College. 

Cost of Enhanced Opening Doors 

Table 5.1 summarizes the short-term (the two program semesters) cost analysis find- 
ings. The Enhanced Opening Doors program components cost $1,623 per program group 
member. The first-semester College Success course made up 58.5 percent ($949 per program 
group member) of the cost associated with the program. The second-semester College Success 
course was also a substantial cost, representing 21.5 percent ($349 per program group member) 
of the program cost. The remaining cost is related to enhanced services that are offered to the 
program group; these include enhanced Success Centers, enhanced counseling, and the program 
textbook voucher. 

Every student at Chaffey is apportioned a basic cost for Success Centers (estimated to 
be $1 10 per sample member); this value is captured in the cost of “Other college courses” and is 
not independently visible in Table 5.1. For Enhanced Opening Doors, a staff person was added 
to coordinate Success Center activities for program group members. On a per program group 
member basis, the cost of this coordinator was $1 19. As a result, the total investment in Success 
Centers ($110 plus $119) is roughly two times greater per program group member ($229) than 
it is per control group member ($110). 

Similar to the cost of Success Centers, the total investment in counseling per sample 
member is not independently visible in Table 5.1. The first $42 of counseling per program 
group member (also not shown in Table 5.1) is captured in the cost of “Other college courses.” 
Student-to-counselor ratios were lowered for the Enhanced Opening Doors program, at an 
additional cost of $147 per program group member, for a total of $189 ($147 plus $42) per 
program group member for counseling. By comparison, the total cost of counseling per control 

4 Moore et al. (2004). 
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The Opening Doors Demonstration 
Table 5.1 

Net Cost of Enhanced Opening Doors at Chaffey College 
During the Two Program Semesters 


Chaffey College Four-Year Follow-Up Report 


Feature 

Program 

Group 

Control 

Group 

Difference 

(Net) 

Program components ($) 

First-semester College Success course 

949 

6 

943 

Second-semester College Success course 

349 

0 

349 

Enhanced services 

Enhanced Success Centers 

119 

1 

118 

Enhanced counseling 

147 

1 

146 

Program textbook voucher 

60 

0 

59 

Subtotal 

1,623 

8 a 

1,615 

Other college courses' 5 ($) 

First semester 

1,296 

1,692 

-396 

Second semester 

1,400 

1,326 

74 

Subtotal 

2,697 

3,019 

-322 

Total (S) 


JWI 



SOURCES: MDRC calculations from Chaffey College transcript and budget data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

Tests of statistical significance were not performed. 

Program costs are based on a steady state of operation that excludes research and start-up costs. 
a One member of the control group received the program treatment in the first semester, so rather than SO, 
program component costs are slightly above $0. 

b Includes the cost of usual services such as counseling and Success Centers. 


group member is $39. 5 As a result, the total investment in counseling is nearly five times greater 
per program group member ($189) than it is per control group member ($39). 

The smallest component of the Enhanced Opening Doors program was the program 
textbook voucher. Approximately 3.7 percent ($60 per program group member) of the program 
cost was associated with textbooks. 


5 The cost of usual services is estimated per credit and is the same for program and control group members. 
Differences in enrollment affect the average number of credits taken per sample member, so the dollar amount 
associated with counseling that is captured in the cost of “Other college courses’’ in Table 5.1 is slighdy 
different for members of the program group ($42) and the control group ($39). 
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In addition to the Enhanced Opening Doors College Success course, program group 
members took other courses at Chaffey. These other courses mostly represent their usual course 
load with a minor offset to fit the Success Course into their schedules. That is, beyond the 
required College Success course, students in the program group attempted an average of 5.5 
other credits in the first program semester and 6.0 other credits in the second program semester. 
For program group members, the cost of these other courses was $1,296 in the first semester 
and $1,400 in the second semester. 

Over the long term (looking at all four years of follow-up), the total cost per program 
group member increases to $9,567 (discussed in more detail later in the chapter). During this 
period, the cost of the Enhanced Opening Doors program components remains unchanged 
from the first two semesters at $1,623. The remaining $7,943 of the long-term cost is associat- 
ed with other college courses attempted at any California two-year public college by program 
group members. 6 

Cost of Usual Services 

In the short term (for the two program semesters only), a typical member of the control 
group had an estimated cost of $3,027, as presented in Table 5.1. Because control group 
members were not eligible to receive the enhanced services that were provided as part of the 
Enhanced Opening Doors program, essentially all of this cost — over 99 percent ($3,019 per 
control group member) — was associated with other college courses. 7 Specifically, students in 
the control group took an average of 7.2 other credits in the first program semester and 5.7 other 
credits in the second program semester (not shown); the cost of these other courses was $1,692 
in the first semester and $1,326 in the second semester. 

Over the long term (looking at all four years of follow-up), the cost per control group 
member increases to $8,541 (discussed in more detail later in the chapter). During the four-year 
follow-up period, the minute cost that was originally associated with Enhanced Opening Doors 
program components remains unchanged from the two program semesters. Nearly all of the 
cost associated with control group members is associated with other college courses. During the 


6 The cost of other college courses is estimated by multiplying the number of credits attempted at any Cali- 
fornia two-year public college by the 2006-2007 cost per credit after discounting and adjusting for inflation. 
Inflation adjustment is based on the Higher Education Price Index inflation estimates for two-year public 
colleges. The discount rate is 3.5 percent. 

7 One control group member received Enhanced Opening Doors services, so rather than $0 in the panel 
showing program component costs, the costs are virtually $0. 
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four years of follow-up, the average control group member attempted $8,535 worth of other 
college courses at any California two-year public college. 8 

Net Cost 

In the short term, as highlighted in Table 5.1, the net cost of the program components is 
$1,615 per program group member. When combined with the cost difference associated with 
other college courses, the total net cost of the two program semesters was $1,293 per program 
group member. Specifically, in the first semester, the cost of other college courses attempted at 
Chaffey College per control group member was $396 greater than the cost per program group 
member. In the second semester, the direction of this difference changed, as the cost of other 
college courses attempted at Chaffey College per program group member was $74 greater than 
the cost per control group member. 9 

As observed earlier, over the long term, when the entire four-year follow-up period is 
considered and when costs at any California two-year public college are included, the cost per 
program group member is estimated to be $9,567, and the cost per control group member is 
$8,541. These estimates demonstrate that the cost for the typical program group member was 
$1,023 more than the cost for the typical control group member during the four years of follow- 
up. During that period of time, the typical control group member attempted two and a half 
credits more than the typical program group member (excluding Enhanced Opening Doors 
College Success course credits); the estimated value of this difference is $592 per program 
group member. 10 This difference offset the net cost of the program components shown in Table 
5.1 ($1,615) and resulted in a long-term net cost of $1,023 per program group member. 


8 The $8,535 is derived by multiplying the number of credits attempted (by year) by the corresponding cost 
per credit. Cumulative credits attempted by year are shown in Appendix Table A.2. 

9 Why is the cost of other college courses for the control group greater than the program group in the first 
semester but not in the second? During the first semester, program group members were told that they had to 
enroll in the College Success course, and, as a result, many of them took fewer other college courses. As 
exhibited by the control group, in the absence of the Enhanced Opening Doors program, a typical probationary 
student in this study would take more non-College Success (“other”) courses if he or she were not mandated to 
participate in the Success Course. However, the decreased enrollment in other college courses is not directly 
proportional to the increased enrollment in the College Success course, because program group members did 
not always decrease their participation in other college courses as a result of their participation in the College 
Success course. In the second semester, program group members were not required to take the College Success 
course, so they returned to a more “normal” level of attempted credits in other college courses. At that point, 
program group members actually began taking more “other” courses than control group members. A portion of 
this difference in attempted credits is attributable to a higher rate of enrollment for the program group. 

10 During the four years of follow-up, control group members attempted more credits than did program 
group members. However, program group members still earned more credits during that time. This difference 
is possible because program group members earned 62 percent of their attempted credits while control group 
members earned 52 percent of their attempted credits. 
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Cost Implications of Alternative Scenarios 

Sensitivity analysis provides information about the degree to which results are sensitive to 
changes in underlying assumptions. Conducting a sensitivity analysis involves varying im- 
portant or uncertain assumptions, and then examining the impact that these changes have on the 
results. This sensitivity analysis adjusts multiple variables in order to answer three questions: 
How much would the net cost of the program change if revenue associated with a higher 
enrollment rate for the program group was considered? How much would the net cost of the 
program change if the cost per Enhanced Opening Doors credit (that is, a College Success 
course credit) equaled the cost of a typical credit (that is, a credit from any course other than the 
College Success course) at Chaffey College? And, how much would the net cost change if the 
second-semester College Success course was dropped from the intervention? 

Higher Enrollment Rate for the Program Group 

• How much would the net cost of the program change if revenue associat- 
ed with a higher enrollment rate for the program group was considered? 

The primary analysis presented in this chapter does not estimate the revenue associated 
with higher rates of enrollment, although during both program semesters the program group had 
higher rates of enrollment at Chaffey than did the control group (as shown earlier in Table 
3.1). 11 As a result, increased enrollment among the program group (a desirable occurrence) 
increased the cost per program group member, making the program more expensive. Assuming 
that students represent both costs and revenues to a college, it is reasonable to offset the pro- 
gram cost with an estimate of the revenue associated with increased enrollment. The revenue 
associated with increased enrollment was $92 per program group member in the first semester 
plus $170 in the second semester. 12 This offset, then, lowers the net cost per program group 
member by $262, which results in a short-term net cost of $1,031 (program cost of $1,293 
minus revenue of $262) and a long-term net cost of $761 (program cost of $1,023 minus 
revenue of $262). 13 


"During the first semester, 85.3 percent of program group members compared with 80.9 percent of con- 
trol group members enrolled in classes at Chaffey. During the second semester, 67.0 percent of program group 
members compared with 59.6 percent of control group members enrolled in classes at Chaffey. These values 
are slightly different from the values shown in Table 3.1 because the second-semester values there include 
summer 2007 data, which are excluded in this chapter. 

12 These values are based on enrollment differences between program and control group members at 
Chaffey College during the two program semesters only. See CCCCO (2009). 

13 The revenue associated with a higher rate of enrollment for the program group is estimated by multiply- 
ing the percentage point difference in enrollment between the program and control groups by the average 
number of total credits attempted (including the College Success course and other college courses) per enrolled 
program group member. Then, that value is multiplied by the revenue per credit that the college receives, 

(continued) 
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Adjusting the Cost of the Program Credit 

• How much would the net cost of the program change if the cost per En- 
hanced Opening Doors credit equaled the cost of a typical credit at 
Chaffey College? 

As noted earlier, enrollment shortfalls caused Enhanced Opening Doors costs to appear 
higher than they might under different circumstances. 14 The estimated cost per College Success 
credit is $590. This is approximately two and a half times the typical cost per credit at Chaffey 
College, which equals $234. If the college is able to make adjustments to the College Success 
course that bring its cost in line with other classes at Chaffey, the short-term net cost per 
program group member would decrease from $1,293 to $317, while the long-term net cost 
would fall from a cost of $1,023 to a cost of less than $50. A cost reduction of this magnitude 
may be too extreme to implement immediately, but the fact that such an adjustment can produce 
such a large shift illustrates that significant cost reductions may be possible. 

Eliminating the Second-Semester College Success Course 

• How much would the net cost change if the second-semester College 
Success course was dropped? 

As implemented, Enhanced Opening Doors was a two-semester program. Program 
group members were told that they were required to enroll in the Student Success course for the 
first semester, which resulted in a high rate of participation (72 percent) and more intensive 
treatment than control group members experienced. Program group students were encouraged, 
but not required, to enroll in the course for the second semester. The second-semester participa- 
tion rate was 29 percent. As a result of this fall-off in participation in the second semester, that 
portion of the program was more expensive per participant because budgeted resources were 
not reallocated in response to actual enrollment. However, following the evaluation, Chaffey 
College chose to drop the second-semester course from the program in light of the low enroll- 
ment during the study period. If the second-semester intervention (that is, the College Success 
course) is dropped from the program, the net short-term cost per program group member 
decreases from $1,293 to $944, and the long-term cost per program group member falls from 
$1,023 to $674. 


which is assumed to equal the collegewide cost per credit. As a result, the revenue per credit is greater than just 
tuition and state aid received per credit. However, roughly two-thirds of Chaffey revenue is estimated to be 
purely determined by formula. Therefore, if the analysis was adjusted so only computational revenue was used 
to estimate the revenue associated with increased retention, then the revenue received per credit would equal 
roughly $153 and the net cost per program group member would be $1,121. See CCCCO (2009). 

1 Enrollment among program group members fell short of the target in the original program budget. 


56 



Cost-Effectiveness 

A cost-effectiveness analysis compares costs and impacts in order to answer the question, “How 
effective was each dollar at changing the outcomes we care about most?’' To illustrate the 
effectiveness of investing in the Enhanced Opening Doors program, this section puts program 
and control costs in terms of cost per outcome. Table 5.2 summarizes the short-term costs and 
short-term outcomes, while Table 5.3 presents long-term costs and long-term outcomes. 
Comparing program costs with key outcome measures helps identify what the program does 
well and what it does not do well. 

Short-term cost-effectiveness compares Chaffey College costs during the two program 
semesters with three key outcome measures: (1) achieving good academic standing, (2) total 
credits earned, and (3) degree-applicable and developmental credits earned. The cost- 
effectiveness of the Enhanced Opening Doors program compared with the standard college 
services changes depending on which outcome measure is considered most important. 

When short-term program costs are considered in terms of students achieving good ac- 
ademic standing, the program is cost-effective because program group members were nearly 
twice as likely to move into good academic standing as were control group members (30.4 
percent compared with 15.9 percent, respectively); this observed difference is statistically 
significant at the 1 percent level. As a result of this difference, the cost per program group 
student to achieve good academic standing during the program year (about $14,200) was 25 
percent less expensive than the cost per control group student to achieve good academic 
standing ($19,037). 

When short-term program costs are considered in terms of total credits earned during 
the two program semesters (that is, including credits from the College Success course, devel- 
opmental education courses, and degree-applicable courses), the program is slightly cost- 
effective because program group members typically earned more credits, which resulted in a 
slightly lower cost per credit earned for program group members compared with the cost for 
control group members. Specifically, the program group cost per credit earned ($516) was 3 
percent less expensive than the control group cost per credit earned ($530). 15 The difference in 
total credits earned is statistically significant at the 1 percent level. 

The cost-effectiveness of earning credits fades away when only degree-applicable and 
developmental credits are considered. Specifically, during the two program semesters, the cost 
per degree-applicable and developmental credit earned for program group members ($797) was 


l5 This difference is driven by earned credits relative to attempted credits: program group members earned 
59 percent of their attempted credits, while control group members earned 44 percent of their attempted credits 
during the program year. 
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The Opening Doors Demonstration 
Table 5.2 


Cost-Effectiveness of Enhanced Opening Doors at Chaffey College 
During the Two Program Semesters 


Chaffey College Four-Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Cost per sample member, first two semesters ($) 

4,320 

3,027 

1,293 

Academic standing 

Ever in good academic standing (%) 

30.4 

15.9 

14 5 *** 

Cost per sample member in good academic standing ($) 

14,210 

19,037 

-4,827 

Cumulative credits 

Credits earned 

8.4 

5.7 

2 7 *** 

Cost per credit earned ($) 

516 

530 

-13 

Cumulative degree-applicable and developmental credits'* 

Credits earned 

5.4 

4.8 

0.7 

Cost per credit earned ($) 

797 

635 

162 

Sample size (n = 444) 

224 

220 



SOURCES: MDRC calculations from Chaffey College transcript and budget data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

For academic outcomes, a two-tailed t-test was applied to differences between research groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Estimates are adjusted by 
round of random assignment. 

Significance tests were not conducted for cost-effectiveness measures. 

Program costs are based on a steady state of operation that excludes research and start-up costs, 
includes credits earned in courses taken during the first two semesters of the study and excludes credits from 
the College Success course. If a student receives a letter grade in a course, this measure only includes credits in 
which the student earned a “C” or higher. If a student takes a class “pass/fail” or if the class is only offered 
“pass/fail,” then this measure only includes a grade of "pass.” 


26 percent higher than the same type of cost for control group members ($635). 16 The difference 
between program and control group members in degree-applicable and developmental credits 
earned is not statistically significant. The program is not cost-effective in the short term when 
costs are compared with degree-applicable and developmental credits earned because the 
estimated impact on this outcome measure was not substantial enough to offset the initial 
investment in the program. 


16 During the two program semesters, the typical program group member earned 5.4 degree-applicable and 
developmental credits, while the typical control group member earned 4.8 such credits. 
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The Opening Doors Demonstration 
Table 5.3 


Cost-Effectiveness of Enhanced Opening Doors at Any Public, 
Two-Year California College, Years One Through Four 


Chaff ey College Four- Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Cost per sample member, first four years ($) 

9,567 

8,541 

1,026 

Cumulative degree-applicable and developmental credits 3 

Credits earned 

18.4 

17.1 

1.4 

Cost per credit earned ($) 

519 

500 

19 

Sample size (n = 444) 

224 

220 



SOURCES: MDRC calculations from Chaffey College transcript and budget and California Community 
College Chancellor's Office data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Program costs are based on a steady state of operation that excludes research and start-up costs. 

The long-term cost is estimated by adding the cost of the program and the cost of all other credits 
attempted. The cost of other college courses is estimated by multiplying the number of credits attempted at 
any California two-year public college by the 2006-2007 cost per credit after discounting and adjusting for 
inflation. Inflation adjustment is based on the Higher Education Price Index inflation estimates for two-year 
public colleges. The discount rate is 3.5 percent. 

“Includes credits earned in courses taken during years one through four of the study and excludes credits 
from the College Success course. If a student receives a letter grade in a course, this measure only includes 
credits in which the student earned a “C” or higher. If a student takes a class “pass/fail” or if the class is only 
offered “pass/fail,” then this measure only includes a grade of “pass.” 


Long-term cost-effectiveness compares costs from any California two-year public col- 
lege with the number of degree-applicable and developmental credits earned during the four 
years of follow-up. In general, long-term impacts were not statistically significant. As a result, 
long-term differences between the program group and the control group may not be a product of 
the intervention and may simply reflect differences that arose by chance. Other long-term 
outcome measures, such as earning a degree or certificate, are not discussed in this section 
because the estimated impacts are not significant, and the outcome levels (for both the program 
group and the control group) are low. 

The program appears to be relatively cost-neutral when long-term costs are compared 
with degree-applicable and developmental credits earned. Specifically, as shown in Table 5.3, a 
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typical program group member earned 18.4 credits at an average cost of $5 19 per credit earned, 
while a typical control group member earned 17.1 credits at an average cost of $500 per credit 
earned. The estimated difference between program and control group members in degree- 
applicable and developmental credits earned is not statistically significant; however, if this 
difference is “real” — that is, not a chance occurrence — then in terms of degree-applicable and 
developmental credits earned, the program is nearly as effective dollar-for-dollar as the general 
college experience without the program. The long-term cost per degree-applicable and devel- 
opmental credit earned, approximately $510 for both program and control group members, is 
lower than the short-term cost per degree-applicable and developmental credit earned per group 
member, which ranged from $635 to $797 (Table 5.2). This decrease in cost per credit over time 
is driven by an increase in the overall rate of earning credits that are attempted. In the first year 
of follow-up, sample members passed 53 percent of all credits attempted at any California two- 
year public college. In the fourth year of follow-up, sample members were passing nearly 64 
percent of all credits attempted at any California two-year public college. 


Implications 

Program Costs at Chaffey College Compared with Other Institutions 

At Chaffey College, the Success Centers were a pre-existing resource, with costs shared 
across the entire student body. For Enhanced Opening Doors, a staff member was provided in 
addition to the usual Success Center services to coordinate activities for members of the 
program group. Many colleges have a resource similar to the Success Centers at Chaffey 
College; however, the cost of mounting a program like Enhanced Opening Doors at a school 
lacking such services would be much higher than was observed at Chaffey College. 

Program Funding and Sustainability 

Chaffey College contributed the majority of funding (62 percent) for the Enhanced 
Opening Doors program. The other 38 percent came from outside grants. The substantial 
contributions from Chaffey made it easier for the college to continue operating the program 
when grant funds ran out. The program is still in operation today; however, it now exists as a 
one-semester, three-credit intervention. The second-semester, two-credit College Success 
course is no longer offered. Additionally, the Enhanced Opening Doors program seems to 
include a number of fixed costs that would not change substantially if the program were 
moderately expanded. For instance, the cost of the Success Centers would not change if 
program participation increased moderately, nor would the cost of a counselor dedicated to the 
program. As a result, expanding the number of students who take the Enhanced Opening Doors 
course would improve the overall cost-effectiveness of the program because fixed costs would 
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be spread across more students. Chaffey has attempted to do this by institutionalizing the 
program as part of the probation process. Another way to increase participation and achieve 
economies of scale would be to mandate participation. Participation levels were substantially 
higher in the first program semester, when students were told that participation was mandatory, 
than they were in the second semester, when participation was optional. 


Conclusion 

The Chaffey Enhanced Opening Doors program had a short-term net cost of about $1,300 per 
program group member and a long-term net cost of about $1,020 per program group member. 
There seem to be multiple reasonable approaches to decreasing these costs. Such changes 
include dropping the second-semester College Success course, adjusting spending based on 
actual enrollment, and including revenue from increased enrollment. 

When the cost is considered in comparison with outcomes, the Enhanced Opening 
Doors program produces mixed results in the short term and was likely cost-neutral over the 
long term. Notably, the program was cost-effective at moving students into good academic 
standing in the short term. Additionally, the cost per earned credit during the two program 
semesters was slightly lower for program group members compared with control group mem- 
bers, but this positive observation is seen only when all credits are considered, and fades away 
when considering only degree-applicable and developmental credits. Finally, impacts on key 
long-term outcomes, such as degree-applicable and developmental credits earned, were not 
statistically significant. As a result, the program was not able to display definitive cost- 
effectiveness over the long term; it seems more likely that the program is simply cost-neutral, 
meaning that over the long term, the cost per key outcome is essentially the same with or 
without the program. 
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Chapter 6 

Summary and Conclusions 


This report presents the key short- and long-term educational effects of the Enhanced Opening 
Doors program at Chaffey College in Rancho Cucamonga, California. The Enhanced Opening 
Doors program targeted a very specific subpopulation at the college: students on probation. 
These are students who, owing to poor grades or inadequate academic progress, were identified 
by the school as being at high risk of not graduating. The program sought to get these students 
back on track by telling them that they were required to take a “College Success” course, taught 
by a college counselor, which provided basic information on study skills and the requirements 
of college. As part of the course, students were expected to visit the college’ s “Success Centers” 
— which were established at Chaffey in response to the school's recognition that many of its 
entering students were not prepared for college-level work, and where students could receive 
supplementary individualized or group instruction in math, reading, and writing. 

This random assignment study of the Enhanced Opening Doors program reports on the 
effectiveness of the program compared with Chaffey College’ s usual services for probationary 
students. It is critical to recognize this fact when interpreting the findings presented in this 
report. Students who were randomly assigned to the control group received Chaffey College’s 
regular services (which are available to all Chaffey students), including the opportunity to take 
classes with Chaffey’ s dedicated faculty, the opportunity to receive advising from Chaffey’ s 
committed counselors, and so forth. Therefore, the impact of Enhanced Opening Doors repre- 
sents the value that the program added above and beyond Chaffey’ s usual services. 

During the year in which the Enhanced Opening Doors program ran, its impacts were 
promising. On average, students in the program outperformed their control group counterparts 
with respect to total credits earned, grade point average (GPA), and good academic standing. 
However, there were indications that longer-term follow-up was necessary to provide a more 
meaningful picture of the program’s effectiveness. For example, while the program had a 
positive impact on total credits earned during the first year of the study — including credits that 
were applicable toward a degree and those that were not — this impact was driven largely by 
the College Success course, which does not offer degree-applicable credits and thus does not 
directly help students progress toward a degree. Similarly, the program’s positive impact on 
GPA during the first year of the study is attenuated (although still positive and significant) when 
the College Success course is not included in the GPA calculation. Nonetheless, the hope of the 
program’s designers is that the habits and study skills developed in the College Success course 
lead to longer-term success in other classes. In addition, program group students who took the 
three-credit College Success course, which required time and effort to complete, did not fall 
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behind control group members in degree-applicable plus developmental credits earned, indicat- 
ing that the program seemed to be on track. The one-year findings show definite promise, but 
also a degree of uncertainty. 

The study team continued to track the sample for four years after students entered the 
study, and those findings are reported here. With four years of follow-up, any positive effect of 
the program would be expected to translate into students making more progress toward a degree 
with respect to degree-applicable credits, which students must earn in order to graduate, as well 
as developmental credits, which students are encouraged to earn before they enroll in certain 
degree-applicable courses. The evidence from this study does not confirm that the program has 
positive, long-term effects (compared with Chaffey College’s regular services) with respect to 
credit accumulation, continued enrollment, or attainment of a degree or certificate. In general, 
four years after students entered the study, the program group’s outcomes are quite similar to 
those of the control group. After four years, program group members maintain their edge in 
total credits earned, but this impact remains a result of the three-credit College Success course 
that students took during the first semester of the study, which does not help move them toward 
earning a degree or certificate. Strikingly, four years after the study began, only 7 percent of all 
students in the study (in both the program and control groups) had earned a degree or certificate, 
and 44 percent of the sample were still enrolled in some school (with 21 to 22 percent still at 
Chaffey College). 


The Opening Doors Program in 2011 

In part as a result of the documented short-term success of the Enhanced Opening Doors 
program and in part in recognition of the college’ s willingness to reflect on the effectiveness of 
one of its own programs, Chaffey College’s Enhanced Opening Doors program was named one 
of three winners of the 2010 MetLife Foundation Community College Excellence Award. This 
prestigious award is a point of pride for staff at Chaffey College, as is the continued Enhanced 
Opening Doors program. 

As of fall 2011, Chaffey College’s Enhanced Opening Doors program continues to op- 
erate, evolve, and grow. In terms of program evolution, several changes have occurred since the 
MDRC study of the program was conducted. First, the target population has changed. The 
Enhanced Opening Doors program described in this report targeted program students in their 
first, second, or third semester on probation. Since that time, the program has focused its efforts 
on “Level 2” probationary students — that is, students who are in their second semester on 
probation. (Level 2 probation students represent about one-fourth to one-third of all students on 
probation, depending on the year or semester.) This change was based on college staff mem- 
bers' belief that not all students require the more intensive Enhanced Opening Doors program; 
in fact, according to an Enhanced Opening Doors coordinator and counselor, many students 
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who are in their first semester on probation manage to regain their good academic standing 
without the program. 

A second shift is that, whereas the Enhanced Opening Doors program tried to operate 
back-to-back College Success courses over two semesters, the program now includes a single, 
one-semester College Success course that combines the two-semester curriculum from the old 
courses. The second-semester College Success course was not well attended during the MDRC 
study, and eliminating this course should produce significant cost savings to the college. 

In terms of program growth, despite limiting the target population to Level 2 probation- 
ary students, the number of students served by the Enhanced Opening Doors program has 
increased since the fall 2006 cohort enrolled. Figure 6.1 plots, from fall 2007 through fall 2010, 
the number of students on Level 2 probation, the number of students who attended an infor- 
mation session on Enhanced Opening Doors, and the number of students who signed a contract 
to participate in the Opening Doors program. The number of students on Level 2 probation has 
oscillated during that time period, ranging from 700 to 900; meanwhile, the number of students 
served by the program has nearly doubled, from just over 200 in fall 2007 to just under 400 in 
fall 2010, when the program served over 50 percent of all Level 2 probationary students. 

In addition to the expansion of the program within Chaffey College, at least one other 
California college is replicating Chaffey’ s Enhanced Opening Doors program. Chaffey’ s 
program staff members have been leading in-service training for staff members at the replica- 
tion college. 

Since the fall of 2006, the Enhanced Opening Doors program at Chaffey has continued 
to expand and evolve in order to attempt to improve the college’s services for its probationary 
students. The college staff members continue to work hard and experiment with new ideas. 
With the long-term results of the Opening Doors study provided in this report, now is a good 
time to explore the options for probationary students in the future. 


Exploring Options for Probationary Students 

This report examines the effectiveness of a program that was designed to improve academic 
outcomes for students on probation owing to poor grades or inadequate academic progress. As 
might be expected, the long-term academic outcomes for these at-risk students are particularly 
low. Four years after this study began, the rate of receipt of a degree or certificate was well 
below 10 percent, and around 45 percent of students in the study were either still enrolled or had 
earned a degree or certificate. Since these students were already on probation when the study 
began, they all had been in school for at least a year; in fact, on average, students in this study 
had been in college, on average, for 3.56 semesters prior to the study’s start. In other words, the 
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The Opening Doors Demonstration 
Figure 6.1 

Enhanced Opening Doors Program Participation, Fall 2007 to Fall 2010 
Chaff ey College Four- Year Follow-Up Report 
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SOURCE: MDRC field research data. 

NOTES: If students have been on academic or progress probation for the last two semesters in which 
they have been enrolled, they are placed on “Level 2 probation.” 

Students on Level 2 probation are required to attend an information session on the Opening Doors 
program before they can register for classes. “Attended information session” refers to the number of 
students who attended this mandatory information session. 


degree/certificate attainment figures are not four-year graduation rates, but are closer to five- or 
six-year graduation rates. This gives reason to pause and reflect on the options that Chaffey 
College and others have for best serving probationary students. Several options are given below, 
although this list is in no way exhaustive. 
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• Place greater emphasis on early intervention. 

As noted above, since the fall 2006 cohort of students experienced Chaffey’s Enhanced 
Opening Doors program, several changes have been made. One change is that the program now 
targets students in their second semester of probation, as opposed to all probationary students. 
As described by a Chaffey staff member, the reason for this change is that many students in 
their first semester on probation manage to get off of probation without a targeted intervention. 
Given the low rates of success among students in this study (including those who began in their 
first semester on probation), it is likely that students in their first semester on probation are not 
successfully completing the requirements for a degree or certificate in a timely fashion, even if 
they successfully move off of probation. As a result, Chaffey and other colleges might consider 
targeting interventions at students who are getting off track even earlier, before they have had a 
chance to dig too deep a hole for themselves. Colleges may want to keep track of indicators that 
students are at risk of dropping out even before they are ever placed on probation. Early 
intervention has become a mantra in early childhood research, and a similar focus has emerged 
in attempts to reduce drop-out rates in high schools, where “early warning indicator systems” 
are popular. 1 Identifying and assisting students who are likely to end up on probation may be a 
good way to prevent students from ever ending up on probation in the first place. Doing so first 
requires identifying key indicators that students are at risk, which can enable the targeting of 
resources to those students before it may be too late. Likely the greater challenge rests in 
knowing what to do once high-risk students have been identified. Unfortunately, there is only a 
limited body of rigorous evidence on program effectiveness, and an even more limited body of 
evidence on programs that lead to increases in degree or certificate attainment, so there is much 
more work to be done on creating, testing, and evaluating potential solutions. 

• Retool the program. 

Through their work on the Enhanced Opening Doors program, staff members at 
Chaffey have developed a national reputation as innovators willing to test out new ideas and 
continually improve upon the parts of their programs that appear to work, while eliminating 
the parts that are less effective. Although this follow-up study suggests the limited long-term 
success (with respect to academic outcomes) of the Enhanced Opening Doors program, 
Chaffey is already seeking ways to better serve its students. The college has eliminated the 
second-semester College Success course, which had limited participation and was relatively 
costly. The college is also working to improve the services offered at the Success Centers, 
including attempting to make “counselor apprentices” available to meet with students, do 
progress and graduation checks with them to make sure they are on track to graduate on time, 


1 Nei Id (2009); www.princeton.edu/futureofchildren/publications/docs/19_01_04.pdf. 
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and instruct students on how they can continue to audit their own progress. In addition, 
instructors in some of the college’s “foundation courses” (for example, developmental reading) 
now require students to spend a certain number of hours in the Success Centers, a relatively 
inexpensive course add-on for a college that already has the centers in place, including a 
system for tracking students’ use. 

While it was expected that the Enhanced Opening Doors program’s College Success 
course and required visits to the Success Centers would lead students to form the habit and 
routine of visiting the Success Center even after the program was complete, this expectation did 
not play out in reality. As a result, if Chaffey staff members believe that visiting the Success 
Centers is valuable for all probationary students, they might consider extending the required 
visits beyond the program semester(s). 

• Be more assertive about dismissing students who are on probation. 

One other option for colleges to consider is being more assertive in dismissing students 
who are on probation. Many factors should weigh into this very difficult decision, two of which 
are the college’ s mission and level of resources. Resources are always limited and are especially 
scarce in the current fiscal climate in a state like California. Colleges must therefore decide 
whether and how many resources they are willing to dedicate to students whose need may be 
the greatest, but whose likelihood of success may be the lowest. A program like Chaffey’ s 
Enhanced Opening Doors program costs around $944 per student as a one-semester interven- 
tion, but, for example, eliminating the College Success course and only requiring visits to the 
Success Centers might yield similar results at a lower cost. These are difficult decisions to 
consider, but ignoring or not acknowledging them has consequences. Perhaps some probation- 
ary students would be better served by other education and training services in the community 
that offer different or more intensive supports, such as the federally funded Job Corps program; 
perhaps others might consider taking time off to resolve any personal barriers that are interfer- 
ing with school. 

For some students, the expected long-term financial benefits of community college 
might no longer be positive when their likelihood of graduating drops to extremely low levels. 
Given the opportunity cost and cost of attendance, 2 at some point a student’s likelihood of 
earning a degree may be low enough that the expected economic return of attending school is 
negative. While the mission of most colleges is greater than economic returns to credit accumu- 
lation and degree attainment, economics may be one factor that influences many colleges’ 
policies for dealing with students who are unlikely to graduate. Of equal importance, infor- 
mation regarding the expected financial returns of continued enrollment may influence students’ 

2 The cost of attendance is so small in California that this is likely not a large factor. 
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interest in continuing to pursue a degree. Unfortunately, little guidance is available to determine 
whether (or at what level) a student’s probability of graduating can be expected to yield a 
negative financial return. 

The above discussion brings up just a few considerations that college staff might want 
to keep in mind as they continue to struggle with the best way to serve probationary students. 


Conclusion 

Chaffey College is one of a small but growing group of colleges in the United States that has 
been willing to subject one of its programs to a rigorous, random assignment evaluation — the 
gold standard design in evaluation research. Because of the college’s commitment to serving its 
students as well as possible, the staff at Chaffey were eager to know how effective the En- 
hanced Opening Doors program was compared with the college’s regular services for proba- 
tionary students. Their willingness to participate in this study should be applauded, as it enables 
them to better serve their students and provides other college administrators, policymakers, and 
researchers with trustworthy information and evidence that can be used to guide decisions. 
Although the evaluation yielded no strong evidence that the Enhanced Opening Doors program 
has long-term outcomes that are significantly different from those produced by Chaffey ’s usual 
services, the college and policymakers can take away some other lessons from this study. 

For example, the Opening Doors study at Chaffey College began as a study of one pro- 
gram (the Opening Doors program), and that program evolved into a stronger second program 
(the Enhanced Opening Doors program). A prior MDRC report describes the differences and 
evolution in detail, but one of the main take-aways in the development of both programs was a 
simple change in language, which seemed to have effects on program participation and short- 
term program impacts . 3 Low rates of participation in the College Success course in the original 
program led Chaffey administrators to require participation in the College Success course in the 
Enhanced Opening Doors program, and they told students that their registration would be 
blocked if they did not comply. This seemingly subtle change in policy, from encouragement to 
requirement, was associated with a large improvement in program participation . 4 Individuals 
who are at the greatest risk of failure may also be the least likely to participate in programs that 
are designed to help them succeed. College administrators must weigh their desire to allow their 
students autonomy and decision-making power against the fact that doing so may reduce 
participation in the very programs that are created for their benefit. Clearly, this a difficult 


3 Scrivener, Sommo, and Collado (2009). 

4 The word “associated,” rather than “caused,” is used here intentionally. The study at Chaffey never ran- 
domly assigned students to an “encouraged to participate” condition and a “required to participate” condition. 
It is possible that other factors led to the observed increase in participation rates. 
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balance to achieve — requiring individuals to participate in a program without their buy-in can 
backfire, yet allowing complete flexibility, especially when dealing with students who have a 
low likelihood of success, may not be in the students’ best interest. 

A final lesson that is emerging from a number of community college studies is that 
one- and two-semester interventions may not be sufficient to make a lasting difference . 5 
Students often do better while they receive interventions, but the impacts fade once the 
interventions end. While it is not realistic — and probably not advisable — for interventions to 
last indefinitely, program operators might consider whether they can do more to help students 
make a smooth transition to regular college services once an intervention like Enhanced 
Opening Doors ends. For example, program operators might place greater emphasis on 
mapping out what courses students should take in subsequent semesters, and conducting 
periodic follow-up to make sure that students remain on track and have not encountered new 
barriers to success. Some experts have suggested that community colleges need to institute 
better policies and create clearer pathways that will help all students earn a certificate or degree 
as quickly as possible . 6 Some promising new initiatives, such as the City University of New 
York’s Accelerated Study in Associate Programs (ASAP ), 7 and the Bill and Melinda Gates 
Foundation’s Completion by Design Initiative , 8 are aspiring to do just that, and may soon offer 
lessons on how this might be achieved. 


5 See, for example. Scrivener and Weiss (2009); Weissman et al. (201 1). 

6 See, for example, Jenkins (201 1); Postsecondary Success Team (2010). 

7 See www.cuny.edu/academics/programs/notable/asap/about.html. 

8 See www.completionbydesign.org/. 
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Appendix A 

Sample Characteristics at Baseline, by Research Group, 
and Supplementary Four-Year Impact Tables 




The Opening Doors Demonstration 

Appendix Table A.l 

Selected Characteristics of Sample Members at Baseline, by Research Group, 

Enhanced Opening Doors Study 


Chaff ey College Four-Year Follow-Up Report 


Characteristic 

Full 

Sample 

Program 

Group 

Control 

Group 

Gender (%) 

Male 

38.3 

38.8 

37.7 

Female 

61.7 

61.2 

62.3 

Age in years (%) 

18-20 

60.6 

60.7 

60.5 

21-25 

29.5 

29.5 

29.5 

26-30 

5.6 

5.4 

5.9 

31-34 

4.3 

4.5 

4.1 

Marital status (%) 

Married 

5.8 

5.8 

5.7 

Unmarried 

94.2 

94.2 

94.3 

Race/ethnicity a (%) 

Hispanic/Latino 

54.2 

54.3 

54.1 

Black, non-Hispanic 

12.2 

13.5 

11.0 

White, non-Hispanic 

21.8 

19.2 

24.4 

Asian or Pacific Islander 

6.0 

7.2 

4.8 

Other 

5.8 

5.8 

5.7 

Has one child or more (%) 

10.8 

8.7 

12.9 

Household receiving any government benefits 15 (%) 

12.1 

8.7 

15.4 ** 

Financially dependent on parents (%) 

51.3 

52.7 

50.0 

Ever employed (%) 

92.6 

94.2 

90.9 

Currently employed (%) 

74.6 

74.5 

74.6 

Diplomas/degrees earned 1 (%) 

High school diploma 

95.7 

95.7 

95.7 

General Educational Development (GED) certificate 

3.6 

2.9 

4.3 

Occupational/technical certificate 

6.3 

6.8 

5.8 

Date of high school graduation/GED certificate receipt (%) 

During the past year 

32.2 

31.3 

33.0 

Between 1 and 5 years ago 

50.9 

51.7 

50.0 

More than 5 years ago 

17.0 

16.9 

17.0 

Main reason for enrolling in college (%) 

To complete a certificate program 

5.8 

5.8 

5.8 

To obtain an associate's degree 

24.8 

23.8 

25.7 

To transfer to a 4-year college/university 

62.9 

64.6 

61.2 

To obtain/update job skills 

2.7 

2.9 

2.4 

Other 

3.9 

2.9 

4.9 


(continued) 
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Appendix Table A.l (continued) 


Characteristic 

Full 

Sample 

Program 

Group 

Control 

Group 

First person in family to attend college (%) 

30.8 

31.1 

30.4 

Working personal computer in home (%) 

90.3 

91.7 

89.0 

Owns or has access to a working car (%) 

88.9 

89.3 

88.5 

Language other than English spoken regularly in home (%) 

35.6 

37.7 

33.5 

U.S. citizen (%) 

93.0 

95.2 

90.9 * 

Respondent bom outside U.S. d (%) 

11.0 

11.2 

10.9 

Respondent or respondent’s parent(s) born outside U.S. d (%) 

50.3 

53.0 

47.5 

Prior academic progress at Chaffev College 1 




Number of semesters enrolled 

3.56 

3.63 

3.49 

Number of credits earned 

13.5 

13.8 

13.2 

Cumulative GPA 

1.4 

1.4 

1.3 

On probation (%) 

98.9 

98.7 

99.1 

One semester on probation 

48.4 

50.0 

46.8 

Two semesters or more on probation 

50.5 

48.7 

52.3 

Sample size 

444 

224 

220 


SOURCES: MDRC calculations using Baseline Information Form (BIF) and Chaffey College transcript and 
probation data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a Respondents who said they are Hispanic/Latino and chose a race are included only in the Hispanic/Latino 
category. Respondents who are not Hispanic/Latino and chose more than one race are considered multiracial. 
“Other” includes American Indian/ Alaskan Native, multiracial, and other. 

b Benefits include Unemployment/Dislocated Worker benefits, Supplemental Security Income (SSI) or 
disability, cash assistance or welfare, food stamps, and Section 8 or public housing. 

distributions may not add to 100 percent because categories are not mutually exclusive. 
d “U.S.” includes Puerto Rico. 

e As of the start of the program. Includes information from Chaffey from fall 1999 through summer 2006. 
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The Opening Doors Demonstration 
Appendix Table A.2 

Credit Accumulation Among Sample Members at Any Public, 
Two-Year California College, Enhanced Opening Doors Study, 
Years One Through Four and at Baseline 


Chaff ey College Four- Year Follow-Up Report 



Program 

Control 

Difference 

Standard 

Outcome 

Group 

Group 

(Impact) 

Error 

Baseline 3 





Credits earned 

16.0 

17.7 

-1.7 

1.3 

Degree-applicable and developmental credits earned b 

13.2 

14.5 

-1.3 

1.1 

Years 1 through 4 





Cumulative credits attempted 





Year 1 

15.9 

15.0 

0.9 

1.0 

Year 2 

25.4 

25.0 

0.4 

1.7 

Year 3 

31.8 

31.4 

0.4 

2.3 

Year 4 

36.7 

36.5 

0.2 

2.9 

Cumulative degree-applicable 
and developmental credits attempted 





Year 1 

13.2 

15.0 

-1.8 * 

0.9 

Year 2 

22.5 

24.8 

-2.3 

1.7 

Year 3 

28.9 

31.1 

-2.2 

2.3 

Year 4 

33.8 

36.2 

-2.5 

2.8 

Cumulative credits earned 





Year 1 

9.4 

6.9 

9 5 *** 

0.8 

Year 2 

15.1 

12.4 

2.7 * 

1.4 

Year 3 

19.2 

16.1 

3.1 * 

1.8 

Year 4 

22.7 

19.1 

3.6 * 

2.1 

Cumulative degree-applicable 
and developmental credits earned 13 





Year 1 

6.4 

5.8 

0.6 

0.7 

Year 2 

11.4 

10.7 

0.7 

1.2 

Year 3 

15.2 

14.2 

1.1 

1.6 

Year 4 

18.4 

17.1 

1.4 

1.9 

Baseline through vear 4 a 





Cumulative credits earned 

38.7 

36.8 

1.9 

2.6 

Cumulative degree-applicable 





and developmental credits earned 13 

31.6 

31.5 

0.1 

2.3 

Sample size (n = 444) 

224 

220 



(continued) 
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Appendix Table A.2 (continued) 


SOURCES: MDRC calculations from Chaffey College transcript and California Community College 
Chancellor's Office data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Credits earned are based on data from fall 2006 through summer 2010. 

““Baseline” measures include all course information and credits from fall 1999 through summer 2006. 
includes credits earned in courses taken during the first four years of the study and excludes credits from the 
College Success course. If a student receives a letter grade in a course, this measure only includes credits in 
which the student earned a “C” or higher. If a student takes a class “pass/fail” or if the class is only offered 
“pass/fail,” then this measure only includes a grade of “pass.” 
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The Opening Doors Demonstration 
Appendix Table A.3 

Percentage of Sample Members Who Enrolled or Earned a Degree or Certificate at 
Any U.S. College, Enhanced Opening Doors Study, Years One Through Four 


Chaffey College Four-Year Follow-Up Report 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Earned a degree or certificate 

Year 1 

1.3 

0.5 

0.9 

0.9 

Year 2 

2.7 

1.8 

0.9 

1.4 

Year 3 

5.8 

5.0 

0.8 

2.2 

Year 4 

7.6 

6.4 

1.2 

2.4 

Enrolled or earned a degree or certificate 

Year 1 

90.6 

90.9 

-0.3 

2.8 

Year 2 

62.0 

63.2 

-1.1 

4.6 

Year 3 

52.7 

46.8 

5.8 

4.7 

Year 4 

45.5 

45.5 

0.1 

4.7 

Sample size (n = 444) 

224 

220 




SOURCES: MDRC calculations from Chaffey College transcript, California Community College Chancellor's 
Office, and National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Enrollment and credits earned are based on data from fall 2006 through summer 2010. Degree and 
certificate data are from fall 2006 through spring 2010. 
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The Opening Doors Demonstration 
Appendix Table A.4 

Primary Academic Outcomes Among Sample Members, by Length of Time on Probation at Baseline, 
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The Opening Doors Demonstration 
Appendix Table A.5 

Academic Outcomes Among Sample Members at Chaffey College, 
Enhanced Opening Doors Study, Years One Through Four 


Chaffey College Four- Year Follow-Up Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Cumulative number of semesters enrolled 

Year 1 

1.53 

1.43 

0.10 

0.07 

Year 2 

2.31 

2.27 

0.04 

0.13 

Year 3 

2.82 

2.73 

0.09 

0.17 

Year 4 

3.15 

3.05 

0.10 

0.21 

Enrolled (%) 

Year 1 

88.8 

88.2 

0.7 

3.0 

Year 2 

47.3 

52.8 

-5.5 

4.7 

Year 3 

32.6 

27.7 

4.8 

4.4 

Year 4 

21.9 

20.9 

1.0 

3.9 

Cumulative credits earned 

Year 1 

9.3 

6.4 

^ Q *** 

0.8 

Year 2 

14.0 

10.8 

3 2** 

1.4 

Year 3 

16.6 

13.3 

3.3 ** 

1.7 

Year 4 

18.5 

14.9 

3.6 * 

1.9 

Cumulative degree-applicable and 
developmental credits earned® 

Year 1 

6.3 

5.4 

1.0 

0.7 

Year 2 

10.4 

9.3 

1.1 

1.2 

Year 3 

12.8 

11.7 

1.1 

1.5 

Year 4 

14.5 

13.2 

1.3 

1.7 

Earned a degree or certificate (%) 

Year 1 

1.3 

0.5 

0.9 

0.9 

Year 2 

2.7 

0.9 

1.8 

1.3 

Year 3 

5.3 

3.6 

1.7 

2.0 

Year 4 

5.8 

5.0 

0.8 

2.1 

Enrolled or earned a degree or certificate (%) 

Year 1 

88.8 

88.2 

0.7 

3.0 

Year 2 

48.6 

53.2 

-4.6 

4.7 

Year 3 

35.3 

28.6 

6.6 

4.4 

Year 4 

26.8 

23.2 

3.6 

4.1 

Sample size (n = 444) 

224 

220 




(continued) 
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Appendix Table A.5 (continued) 


SOURCE: MDRC calculations from Chaffey College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by round of random assignment. 

Enrollment and credits earned are based on data from fall 2006 through summer 2010. Degree and certificate 
data are from fall 2006 through spring 2010. 

“Includes credits earned in courses taken during the first four years of the study and excludes credits from the 
College Success course. If a student receives a letter grade in a course, this measure only includes credits in 
which the student earned a “C” or higher. If a student takes a class “pass/fail” or if the class is only offered 
“pass/fail,” then this measure only includes a grade of "pass.” 
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Appendix B 

Measure Creation 




This appendix gives details on how academic outcome measures used in this report were creat- 
ed. In particular, the process of combining data sources and the definition of time periods, such 
as semester and year, are described. These data processing decisions pertain most closely to the 
academic outcomes measures shown in Chapters 3 and 4 of this report. 


Combining Data Sources 

As described in Chapter 2, MDRC received academic outcome data for program and control 
group students from three data sources: Chaffey College, the California Community College 
Chancellor’s Office (CCCCO), and the National Student Clearinghouse. The scope of schools 
covered and data available vary by data provider. 

Chaffey supplied MDRC with student-level transcript, 1 student probationary status, 
Success Center participation, and degree receipt data for courses taken and degrees or certifi- 
cates earned at Chaffey College. MDRC received similar data from CCCCO, including student- 
level information on courses taken, credits earned, and grade received, as well as de- 
gree/certificate receipt at any participating two-year public California college (of which Chaffey 
College is one). Finally, the National Student Clearinghouse provided enrollment data, not in- 
cluding student course-taking information, and degree/certificate data for students attending any 
of more than 3,000 colleges, 2 encompassing most two-year public California schools. Thus, all 
three data sources cover student academic progress at Chaffey College, and two of three data 
sources provide student progress information at other two-year public California colleges. (See 
Table B.l for a summary.) 

The overlapping nature of these data forced a decision about which data to use when 
discrepancies across sources occurred. Although data comparisons showed remarkable con- 
sistency between sources, some differences did appear. A small subset of these discrepancies 
resulted from students’ absences from a data set. 3 Other inconsistencies occurred as a result of 
the fluidity that is inherent in academic data, as students withdraw from courses after the 
add/drop period, incomplete grades are updated subsequent to the end of the term, and degrees 
are wi t hheld until students complete their nonacademic (paperwork) requirements even if they 


’Transcript data include courses for which students registered, number of credits earned, and course 
grades. 

2 National Student Clearinghouse data cover 90 percent of college enrollment. See 
www.studentclearinghouse.org. 

3 Matching through student identification number resulted in all students being located (that is, having one 
transcript record or more) in both Chaffey and CCCCO data files. National Student Clearinghouse data were 
available for 97.3 percent of the sample. 
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The Opening Doors Demonstration 
Appendix Table B.l 

Data Source Characteristics, Enhanced Opening Doors Study 
Chaffey College Four-Year Follow-Up Report 


Characteristic 


Data Source 


Chaffey College 

California 
Community College 
Chancellor's Office 

National Student 
Clearinghouse 

Covered colleges 




Chaffey College 

V 

V 

V 

California 2-year public colleges 


V 

S 

All U.S. colleges 3 



V 

Data available 




Probation status 

V 



Success Center participation 

V 



Course-level transcript information 13 

V 

V 


Enrollment 0 

s 

V 

V 

Degree/certificate attainment 


V 



SOURCES: MDRC data sources summary using Chaffey College transcript, California Community College 
Chancellor's Office, and National Student Clearinghouse data. 

NOTES: “National Student Clearinghouse data include data for more than 3,000 colleges covering 90 percent of 
college enrollments. See www.clearinghouse.org. 

b Course-level transcript data include courses for which students registered, course name and type, course 
grade, and credits attempted and earned. 

“Enrollment status using Chaffey transcript and California Community College Chancellor's Office data is 
determined using course-level transcript information. 


have completed their academic (coursework) requirements . 4 The imperfect process of assigning 
National Student Clearinghouse enrollment dates to term of enrollment, described in more detail 
below, resulted in some additional differences. 

In general, data from Chaffey College were determined to be the most up-to-date of the 
three sources; whereas schools provide “snapshots” of their data to CCCCO and the National 


different data sources sometimes differ on how they treat people who have completed all the aca- 
demic requirements but not all the nonacademic ones. 
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Student Clearinghouse a few times a year (often without updating previous data submissions), 
the Chaffey College data system is consistently updated with grade, enrollment, and degree re- 
ceipt changes. Similarly, the additional course-taking and degree receipt information available 
through CCCCO make these data more complete than the National Student Clearinghouse data, 
with regard to enrollment and degree receipt at two-year public California colleges. 

As a result, credit accumulation, enrollment, and degree receipt at Chaffey College are 
determined using data provided by Chaffey only. CCCCO data contribute to measures of credit 
accumulation, enrollment, and degree/certificate attainment when pertaining to academic out- 
comes at two-year public California colleges other than Chaffey College. Finally, enrollment 
and degree/certificate receipt at private, for-profit, and four-year colleges, as well as colleges 
outside of California come from data provided by the National Student Clearinghouse. Credit 
accumulation data at these schools are unavailable since the National Student Clearinghouse 
does not include these data. In summary, Chaffey data take precedence over CCCCO data, 
which, in turn, trump National Student Clearinghouse data. 


Timeframes for Measure Creation 

Not only are many data sources combined to create academic outcome measures in this report, 
but also a variety of timeframes related to those measures are presented. The remainder of this 
appendix discusses how terms are assigned and different time periods are defined. 

First, CCCCO degree data and National Student Clearinghouse enrollment and degree 
data were not associated with a term; consequently, terms were assigned based on dates of en- 
rollment or degree award date. Assigning a term to degree/certificate attainment is simpler, 
since only one date is associated with each outcome. For both data sources, credentials awarded 
between October and March were assigned to the fall term, and degrees awarded between April 
and September were assigned to the spring term. 

Assigning term of enrollment to National Student Clearinghouse records is more com- 
plicated. All National Student Clearinghouse enrollment records of more than two weeks (14 
days), the typical length of the add/drop period, were assigned to a term. In general, enrollments 
between January 1 and May 31 were considered spring enrollments; between June 1 and August 
15, summer enrollments; and between August 16 and December 31, fall enrollments. 5 

For the most part, students were enrolled (and earning credits) during fall, spring, and 
summer terms; however, a few students (typically those attending quarter schools) have winter 


5 While this general rule applies, other factors contribute to these decisions, such as the school's calendar 
system and reporting patterns, as well as the registration type (for example, “full time,” “half time,” and “with- 
drawal”) associated with the enrollment record. 
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term enrollments. 6 In order to simplify the presentation of outcomes, fall and winter terms are 
pooled together when credit accumulation and registration are presented. Similarly, enrollment 
and credit accumulation data from spring and summer terms are combined. 

Further simplifying the presentation of academic progress, many measures display year- 
ly outcomes. Since program participation began during the fall 2006 semester, all yearly 
measures begin with the fall term and include the subsequent winter, spring, and summer 
terms. 7 In interpreting “cumulative number of semesters enrolled” measures, note that in one 
year the maximum number of semesters a student could be enrolled, even when concurrently 
enrolled in multiple schools, is two semesters (fall/winter term and spring/summer term). 


6 Quarter schools generally have three 10-week terms (in addition to summer), while semester schools 
generally have two 15-week terms (plus summer). 

7 The “program year” is defined as the fall 2006 and spring 2007 terms only. Year 4 credit accumulation 
measures and enrollment at Chaffey College and other two-year public California colleges are complete. Some 
enrollment data at other colleges may be incomplete. Degree measures included degrees awarded prior to June 
30, 2010. 
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Appendix C 

Statistical Model for the Impact Analysis 




The basic strategy for the impact analysis is to estimate the difference in outcomes between the 
program and control groups, adjusting for blocks, or rounds. (Random assignment of students 
was conducted separately for two rounds of students. The first round targeted students who 
were on probation at the end of the fall 2005 semester, and the second round targeted students 
who were on probation at the end of the spring 2006 semester.) This strategy generates esti- 
mates of the average impact of the opportunity to participate in the Enhanced Opening Doors 
program (the “intent-to-treat”)- 1 A linear model is used to estimate the impact of the program on 
outcomes: 

(1) Vi = 1,1 = i cc h block hi + p x ti + e t 
Where: 


y t = An outcome (for example, credits earned) for student i. 

block bi = A dummy indicator equal to 1 if student i is in random assignment 

block b , and 0 otherwise. There are two blocks. 


tj = A treatment indicator equal to 1 if student i was assigned to the 

treatment group, and 0 otherwise. 


The main coefficient of interest is /3 1 , which reflects the estimated average program im- 
pact. A two-tailed t-test is used to assess whether (l 1 differs from zero. For those primary out- 
comes that are binary, a logistic regression model was used to confirm the findings from the 
linear probability model. As is typically the case in experimental analyses, the results for the 
two types of models, in terms of statistical significance, were the same in all cases. 


'in an “intent-to-treat” analysis, comparisons are made between those who were randomly as- 
signed to the program and control groups, regardless of whether they actually participated in the program 
or, for the control group, in regular classes and services. For a more detailed discussion of intent-to-treat 
analyses, see Bloom (2006). 
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